learning - Academic Conferences Limited

Anne Jelfs and Chetz Colwell
distance education we realise that all universities face similar issues. That is providing good electronic
support to staff and students and therefore evaluating how students are working with module
materials and online provision is paramount.
2. Evaluating internet based educational resources
As a relatively new form of educational delivery there is no clear definition of how education resources
presented electronically can to be evaluated. To feed into the development of our framework we
looked at current practices and the accepted methods of data collection for evaluating electronic
materials. Our methods are based on valid evaluation practices, but our findings inform our
framework.
A number of recommendations relating to the evaluation of interactive systems have been established
by researchers (see Hix & Hartson, 1993; Nielsen 1994; Newman & Lamming, 1995; Joynes, 2000;
Sheard & Markham 2005; Light, 2006). For example, the notion that participants should be recruited
from the target population of users and that participants should be observed as they complete a set of
pre-planned representative tasks (van Schaik, 1999). In fact, there is an assumption that a
comprehensive evaluation of a web-based environment needs to consider both the technical and the
pedagogical aspects of the system (Sheard & Markham 2005).
A major challenge for designers and human computer interaction researchers is to develop software
tools able to engage novice learners and to support their learning even at a distance, because poorly
designed interface which makes students feel lost, confused, or frustrated will hinder effective
learning and information retention (Ardito, Costabile, DeMarsico et al 2006). Ihamäki & Vilpola (2004)
say that if the interface of a VLE is not usable, the user’s focus on the actual content is diminished,
because using the VLE requires a considerable amount of concentration. Good usability, on the other
hand, allows the user to focus on the content thus improving learning results and making websites
accessible has growing importance as the Intranet becomes the main delivery channel for eLearning
resources (Phipps & Kelly 2006).
Often educational evaluations focus on learning outcomes, however with the growth in online
provisions, our research focuses on making the system easy to use and to improve access to high
quality education. In the next section we consider the recognised forms of usability and accessibility
evaluations before moving to the methods we have adopted in our joint work.
One of the methods at the forefront of software evaluations is verbal protocol which is also know as
think aloud protocol This is where a participant is asked to speak about what they are doing and why.
The think aloud protocol requires the participant to complete a set of tasks using a prototype website
and to ‘think aloud’. To say everything they are thinking while they complete the tasks (Turnbow,
Kasianovitz et al 2005). The participant needs to maintain a commentary about what they are doing
and why they are taking a particular path. This allows the researcher to have some understanding of
the choices the participant has made.
It is also valuable if the participants are observed and video recorded so that the participants’ actions
can be used in further data analysis (Hughes & Parkes 2003). Usually there are two observers where
one observer notes everything the participant says and if and how the participant completes the task.
Another observer directs the participant and answers any questions asked. This type of user test
provides essential real-time feedback on potential problems in the design and organization of a
website. It is useful because it gives the researcher immediate information that may be forgotten by
the participant at the end of the evaluation session. An approach we adopt is to have chunks of tasks
so the students are only talking aloud for short periods within the longer session.
When conducting think aloud sessions Dumas (2001) considers there to be three levels of interaction.
Level 1 of verbalization expects the person conducting the research to sit behind and out of view of
the participants and not to speak to the participant unless the person does not follow the instructions.
If participants stop talking they are prompted to keep on talking. This is usually used in problem
solving tasks. Level 2 is the manipulation of non-verbal information such as geometric shapes and
Level 3 participants are asked to verbalise but also to explain why they are thinking or doing
something. It is at Level 3 we usually work, but have on occasions tried Level 1, but our work is
primarily not problem solving, but understanding how we can improve on the current provision.
327