learning - Academic Conferences Limited

Objectives
More efficient and
faster preparation and
transmission of data
for analysis by the
different roles involved
in administration,
management and
quality assurance.
6. Conclusion and next steps
Tim Cappelli
Evaluation Impact
The exams data was sent from the base
hospital to MEO on the day of each
OSCE exam. It was sent electronically
over a secure link rather than as bundles
of paper forms, reducing the time, cost
and security of data transmission back to
the centre.
Because the data is already in
electronic format, the data can be easily
collated and assimilated into existing
systems for analysis and distribution.
Having business rules in the software
that instantly flag incorrect forms or
missing data before the examiners
leave the site reduces the number of
errors and facilitates the QA of the
process.
The pilot of the digital pens was evaluated and reported on to the Senior Management Team of the
School together with a set of recommendations for large-scale roll out across all the base hospital.
The recommendations were:
Clarification of the data requirements of the Medical Exams Office (MEO).
Demonstration of the process of using the pens to MEO staff.
Assessment of the value of providing timely feedback to students: Another pilot, similar to the
first, should be run. This time the results from the pens, including the textual feedback, should be
passed directly to the students once the data has been collated, checked and approved for
distribution.
Test the viability and reliability of the mobile phone upload.
Exploration of vendor support: Although Ubysis provided adequate support and advice prior to,
and during, the pilot, it would be prudent to seek assurances that this level of support will continue
in any future
Training of Base administration staff in downloading and checking the pen’s content.
Training of OSCE examiners: Some examiners expressed concerns over lack of training to
provide appropriate feedback.
The first two points were particularly important as there had been a great deal of resistance to the
introduction of the technology from the MEO. This was partly due to a lack of communication and
resulting misunderstanding of the technology and partly due to concerns about the technology making
certain roles redundant. The result was that despite the pilot, MEO staff were still opposed to the
technology and promoted the view the pilot had been unsuccessful. This ‘dis-information’ rapidly
became the prevailing view and work was required to assure the concerns of the MEO staff. This
demonstrates the importance of embracing all stakeholders at the start of any technology change.
At present, the SMT are reviewing the results of the pilot, together with other initiatives in student
feedback, to determine the most appropriate way forward. A final decision will be based on available
resources and the views of staff and students.
Acknowledgements
Thanks to Dr’s Hilary Dexter and Lucie Byrne-Davies from the Manchester Medical School for their
work on the student evaluation that informs much of this study.
References
Buzzetto-More, A. and Alade, J.A. (2006) “Best Practices in e-Assessment”, Journal of Information Technology
Education, Vol 5, pp 251-269.
Gibbs, G. (2010) Dimensions of Quality, The Higher Education Academy,
http://search3.openobjects.com/kb5/hea/evidencenet/resource.page?record=12nH2AFIYcc [accessed 12
August 2011]
Harden, R and Gleeson F (1979) “Assessment of clinical competence using an objective structured clinical
examination (OSCE)”, Medical Education, Vol 13, No.1, pp39-54.
Love, T and Cooper, T (2004) “Designing online information systems for portfolio-based assessment: Design
criteria and heuristics”, Journal of Information Technology Education, Vol 3, pp65-81.
98