learning - Academic Conferences Limited

Tim Cappelli
forms were processed and the data (pdf copies of all the forms and a csv file of all the scores) made
available to the Medical Exams Office (MEO) over a secure link.
Prior to every OSCE cycle, all examiners attend a compulsory briefing session on how to complete
the forms. During the pilot, a single slide was added to this briefing to inform examiners about the
technology and that the pens would be collected during the break. They were not required to change
their practice in any way.
5. Evaluation
The pilot was observed and key points recorded throughout by the development team. In addition,
participating examiners were given questionnaires to complete and examination and support staff
were interviewed two weeks after the event.
5.1 Feedback from stakeholders
Examiners: 54 examiners took part in the pilot and 42 completed the evaluation questionnaire (78%).
91% of examiners felt that using the pens was the same as normal OSCE exams and 7% felt they
were better. 50% of examiners felt positive and 36% neutral about providing written feedback for all
students. Two examiners (5%) reported feeling displeased about this citing extra burden for
examiners and questioning whether the purpose of an OSCE exam was to provide teaching for the
students. There was also a desire for more information about the technology i.e., what it is for, how
accurate it is. In terms of using the pen, examiners commented that, apart from being a bit bulky and
therefore perhaps altering the readability of their handwriting, the pen was just like using any other
pen.
Administrators: Examination support staff reported that the examiners were positive but surprised
when they were given the briefing in the morning. They reported that there seemed to be initial
concern from the examiners but that the examiners appeared reassured when it was just a ‘pen and
paper’. Staff felt that they needed two more members of staff on exam days than usual for docking the
pens and checking the mark sheets as they docked. However, this would not be the case if the whole
circuit had been completed using pens, since pen docking and form checking occurred at the break,
when staff traditionally collect and check the hardcopy forms. Staff did express enthusiasm to trial the
mobile connection as this would decrease their workload further (no need to collect and dock the
pens) and would allow checking to be done whilst the OSCE was ongoing. Exam staff reported that
instead of sending marks to central exams office on the Tuesday following a week of OSCEs, they
were able to send marks at the end of each day of exams. This would speed up the data analysis
and would enable data analysts to have extra time to perform statistics prior to the exams board.
5.2 Achievement of objectives
Pulling together the observed data and feedback from stakeholders, the outcomes of the pilot were
assessed against the original objectives. These are shown in Table 2.
5.3 Return on Investment
Given the potentially high costs of implementing the digital pens across all the hospital bases, it was
decided to complete a Return on Investment impact analysis, using the potential efficiency savings in
time and resources identified in the pilot. This was designed to sit alongside the evaluation of the
objectives in order to demonstrate the period of time required before any financial savings on the
initial investment were realised. Although financial savings were not seen as a driving force for the
pilot, any evidence of efficiency savings could be used to justify the large initial outlay required for
technologies to be adopted.
The impact analysis was carried out using the JISC InfoNet Impact Calculator
(http://www.jiscinfonet.ac.uk/impact-calculator) which allows articulation of the business processes,
the nature of the benefits to be achieved, the capture of performance data for each of the benefits and
recording of the costs associated with realising the change. The calculator was completed, focusing
on the savings in staff time and resources afforded by the adoption of the technology across the
hospitals. Given the large number of forms involved, the few minutes saved in checking each form
and the few pence saved in not photocopying each form soon accrued to a substantial amount.
However, the largest savings came from no longer having to transport the forms manually from each
96