Guenter Hilmer - Phd Thesis - Glamorgan Dspace - University of ...

5.2 Design of the interview evaluation 5 EVALUATION METHODOLOGY
• Creation of a QA-TLS
The first task the test candidates were asked to undertake was the creation
of a QA-TLS on paper. The creation of the QA-TLS was seen as giving the
evaluation participants a first impression of the usage of the 3-Layer-Model.
The exact task and the material can be found in the appendices in Appendix
A.1 and in Appendix A.2.
• Theoretical conversion of the QA-TLS from a Multiple-Choice-QA into a
True-or-False-QA
Another important issue of this evaluation was to show the participants the
possibility of reusability of once authored material in different ways. This
reuse was done in the simplest way by using different display styles for a
specific TLS.
• Creation of another TLS (lecture)
The next logical step of testing the framework was to ask the candidates to
create a TLS by themselves. Their capability of creating TLSs individually
is another important indicator of their understanding the 3-Layer-Model.
• Linking the created TLSs together
Finally, after authoring and creating new material based on the 3-Layer-
Model it was necessary to ask the candidates to link the different TLSs
together to form a first EM.
After fulfilling the tasks in the manual markup part of the evaluation session the
test participants were asked to give the mid-test interview so that information
about the test participants’ understanding of the 3-Layer-Model could be gained.
Mid-test interview
The mid-test interview is fitted into this evaluation to get a structured feedback
about the manual markup task from the evaluation candidates. During the mid-
test interview the participants were asked the following questions:
136