learning - Academic Conferences Limited

e-Assessment Using Digital Pens – a Pilot Study to
Improve Feedback and Assessment Processes
Tim Cappelli
University of Manchester, UK
Timothy.cappelli-2@manchester.ac.uk
Abstract: Manchester Medical School is the largest medical school in the UK, with over 2000 students on the
MBChB programme. During the final three years of the programme all students undergo regular assessments of
their clinical skills through a series of Objective Structured Clinical Examinations (OSCEs). The OSCEs require
students to carry out a series of simulated exercises in front of an examiner. The examiner completes a scoresheet
for each student, giving a mark between 1 and 7 for each of four criteria together with a ‘global mark’, again
between 1 and 7. The examiner also leaves a small piece of written feedback on the bottom of each form.
Following the exam, all the score-sheets from each of the four teaching hospitals attached to the University are
scanned using an optical reader. This involves a large amount of effort and provides many opportunities for
errors. Due to the work involved and logistical problems, student feedback from the OSCEs is currently limited to
a single mark. Despite the examiners providing the piece of text on the score-sheet, this is only made available to
students scoring less than 4 on their global mark. The students and the School are increasingly motivated to
allow all students access to the written feedback. Hence, in an effort to increase efficiency of the OSCE process
and enable the delivery of student feedback, the Medical School has piloted the use of digital pens as a method
of capturing and processing scoring and feedback. This case study presents the process and evaluation of the
pilot. The study examines the choice of technology, the aims of the pilot and an evaluation of the technology to
assess whether objectives have been achieved. An impact analysis of the use of the pens over a five year period
also shows the return on investment.
Keywords: e-assessment, digital pens, feedback
1. Background
Manchester Medical School is the largest medical school in the UK, with over 2000 students across a
five year MBChB programme. The final three years of the programme are spent based at one of five
teaching hospitals across the North-West of England. With so many students dispersed across so
many sites, the logistics of curriculum delivery and examination is immensely challenging. This is
particularly so when it comes to the assessment of students’ clinical skills. In common with most
medical schools, a student’s clinical skills are assessed using a series of Objective Structured Clinical
Assessments (OSCE) (Harden and Gleeson 1979). OSCEs have been used for over three decades
as a way of providing a reliable and objective method for testing students’ clinical abilities. OSCEs
consist of series of short tests – or stations – that are designed to test a student’s clinical performance
along with competences in history taking and clinical examination. Each of the stations has a different
examiner and students rotate through the stations, to complete all the stations on their circuit. In this
way, all candidates take the same stations, with the same set of examiners, providing the basis of an
objective, structured assessment. The students at the University of Manchester undertake OSCEs
twice a year from year three onwards, with each of the four base hospitals delivering 16 stations per
circuit, 4 times a day over a week to assess all the students in a year group. Each examiner has one
form for each of the students; so with 16 stations and 16 students per circuit, four circuits per day per
hospital and four hospitals, it’s clear that the number of forms soon mounts up. At present all the
forms are marked, collected and checked manually and the forms returned to the Medical Exams
Office (MEO) at the University for scanning and collating. The opportunities for making this process
more efficient and secure through the introduction of electronic marking are clear; several studies
show that e-assessment improves information management and ‘avoids the meltdown of paper
systems’ (Buzzetto-More and Alade 2006, Ridgeway, McCusker and Pead 2007). However, efficiency
savings where not the driving-force behind the Medical School’s commission of a pilot scheme in June
2010; instead it was the lack of feedback given to students after the exam. Each form completed by
the examiner consists of a series of criteria on which the student is given a score between one and
seven, with four representing the minimum standard required. The examiner then gives a ‘global
mark’ for the student’s performance, again between one and seven, and is encouraged to write one or
two lines of feedback in a text box at the bottom of the form. However, due to difficulties in making the
manual forms available to students, only those students who have scored less than four on any given
station are provided with this feedback, and then only sometime after the exam. This is clearly not in
keeping with best practice for summative feedback, with most authors agreeing that feedback should
be constructive, timely and freely available (Nicol and Macfalane-Dick 2006, Gibbs 2010). Increasing
91