learning - Academic Conferences Limited

Tim Cappelli
It was closest to the examiners’ current way of working so does not require acceptance of new
technology and training is negligible.
If the technology fails for some reason, there is always a backup in the form of the paper forms
that have been completed.
The technology is lightweight, simply stored and maintained.
The solution is scalable with no additional cost for software on multiple servers.
The cost of the solution overall is the lowest.
Based on the team’s recommendation, the Medical School went ahead with a pilot of the digital pens
and forms in November 2010.
4. The pilot
It was decided to run the pilot with the selected technology at one of the base-hospitals in order to
assess the viability and practicalities of the pens, how well the stakeholders responded and how well
the devices fulfilled the original objectives.
The team invited proposals from a number of digital pen providers. From a series of email exchanges
and follow up interviews, Ubysis Technologies (http://www.ubisys.co.uk/) were chosen based on their
experience in the Health Service and their accompanying software, Formidable. Formidable is a
server-based application that lets users design, create, print and manage digital forms. It also
interacts with the pen-software, Pen Pusher, to enable the upload and automatic checking of forms
prior to exporting the results to Excel or another data analysis tool. This end-to-end support appeared
to offer the best solution in supporting the OSCE process and a pilot was agreed.
The pilot consisted of an initial dummy run of the pens using a number of dummy OSCE forms
provided by the School and created in Formidable by Ubysis. Members of the development and
OSCE administration teams played the roles of students and examiners. The purpose of the dummy
run was to identify any issues or pressure-points in the process that might occur in the real exam, but
also to reassure OSCE administrators that the technology was reliable and fulfilled their requirements.
Following the dummy run, the School decided to go ahead with a pilot on a real OSCE. Year four
OSCEs were chosen at one of the hospitals and it was decided to run the pilot across 8 stations, with
the other 8 using traditional pens and forms to act as a direct comparison. The pilot would stretch
across 4 cycles per day for 4 days; in total, 1024 forms would be marked using digital pens. The
Medical School purchased 10 digital pens (8 for the pilot and 2 back-up) along with the necessary
licences and a ‘shelf’ of printing patterns that was required to print the required number of unique
forms.
The OSCEs took place in December 2010, and during November, Ubysis worked with the
development team to create and print the forms, train staff on the use of Formidable and ensure all
the technology was ready prior to the OSCEs. They also provided a member of staff to be present on
the first day of the pilot in case of any software or hardware issues occurring.
For the pilot, Formidable was installed on a development server, and two network-connected laptops
were placed in an IT room where the exams took place. Both laptops had docking devices linked via a
USB port, in order to upload the data from the pens to the server. Although it is possible to transmit
the completed forms directly from the pen to the server using a linked mobile phone, it was felt this
would result in additional cost, more confusion for the examiners and a higher risk of data loss.
Instead, it was decided the forms would be downloaded via the laptops at each break, after which
they could be checked for completeness both automatically by the software, and also by the
administrators as a fail-safe.
Forms downloaded automatically once the pens were placed in the docking device and an image of
the mark sheet, alongside a table of the marks gathered from the form, then appeared on the screen.
The software automatically checked for any missing marks and for the presence of text in the
feedback field if the overall score was less than 4. A visual inspection by the administrators ensured
that wherever there was more than one mark for any criteria, that the intended score had been
accepted. Following these checks, each form was saved on the server. At the end of each day, all
95