learning - Academic Conferences Limited

Anne Jelfs and Chetz Colwell
We work together taking the role of observer and facilitator alternatively depending on whether the
person is disabled when Chetz takes the role of facilitator, or non-disabled where Anne is the
facilitator. This method almost replicates the approach suggested by Craven and Booth (2005)
‘Ideally the session should include a facilitator and an observer.The facilitator sits next to
the participant and directs the testing while the observer watches the process remotely
either through a one-way mirror, or by using a video to record the session which the
observer watches in another room. This method enables the observer to concentrate fully
on what the participant is doing and saying, rather than on the actual running of the
experiment.’ (pp 187-188).
While participants complete the tasks, they are observed and prompted as appropriate. We use
probing questions to uncover reasons for particular actions and perceptions of what the participant
expects to find. Tamler (2001) suggests that a good example of a question is ‘when you were
selecting that file what were you expecting’ rather than ‘why did you select that file’ which makes the
user feel that they need to justify their behaviour. We also give some feedback as the student
progresses because they are working as co-evaluators and not purely in isolation so we might say
something like ‘this is just the sort of thing we need to hear or your information will be very useful to
the developers.
Notes are taken by the observer concerning critical incidents, participants’ comments and relevant
actions. Participants are not given a time limit to complete each task. Finally we use a pre-prepared
interview schedule at the end of the session to find out about their opinions and preferences
concerning the websites and to allow them to give any other views they might have. The interview can
also be seen as a de-brief where we might explain further about the work we are conducting.
4. Analysis and report
We do task based analysis – how users tackled the tasks and where the major difficulties arose. The
observer’s notes are invaluable in guiding the analysis as one hour of video tape has been reported to
take at least 5 hours of analysis, or even a day or more (Preece et al 1994, p620). From both the
observer’s notes and the videos we are then able to write a joint report for the developers.
5. Conclusion
This method of working together has proved invaluable to the developers as it give them a report
where they can apply our recommendations knowing that we have discussed our findings and we are
recommending the optimum solution for all users.
If points remain which are inaccessible then these are highlighted, so that statements can be made on
the module website. It is imperative that disabled students are made aware of any potential areas that
are inaccessible for particular types of student or supporting software.
References
Ardito, C., Costabile, M. De Marsico, R. Lanzilotti, S.,Levialdi, T., Roselli, V. Rossano (2006) An Approach to
Usability Evaluation of e-Learning Applications.
http://www.di.uniba.it/~ivu/papers/2006/UAIS2006_Arditoetal.pdf (accessed 09.06.11)
BS 8878: Web accessibility. Code of Practice
http://shop.bsigroup.com/en/ProductDetail/?pid=000000000030180388 (accessed 09.06.11)
Byerley, S. and Chambers, M.B. (2002) Accessibility and usability of web-based library databases for non-visual
users. Library Hi Tech vol 20: 2, pp169-178
Colwell, C., Jelfs, A. & Mallett, E. (2005) Initial requirements of deaf students for video: lessons learned from an
evaluation of a digital video application. Learning, Media and Technology, Vol. 30, No. 2, July, 201–217
Cooper, M., Colwell, C. and Jelfs, A. (2007) ‘Embedding accessibility and usability: considerations for e-learning
research and development projects', ALT-J, Vol 15, No. 3, 231 – 245
Crowther, M.S., Keller, C.C. and Waddoups G.L. (2004) Improving the quality and effectiveness of computermediated
instruction through usability evaluations. British Journal of Educational Technology Vol 35: 3 pp
289–303
Hertzum, M. Hansen, K.D. and Andersen, H. (2009) Scrutinising usability evaluation: does thinking aloud affect
behaviour and mental workload? Behaviour & Information Technology 28:2 pp 165-181
HEFCE (2010) Student perspectives on technology – demand, perceptions and training needs. Report to HEFCE
by NUS
Hix, D. and Hartson, H.R. (1993) Developing user interfaces: Ensuring usability through product and process
Wiley & sons, New York
330