Assessing and reporting performances on pre-sessional EAP courses

J. Banerjee, D. Wall / Journal of English for Academic Purposes 5 (2006) 50–69 61
stage of developing <strong>and</strong> reviewing this new assessment format, we investigated the validity of
the exit assessment in two phases:
† Content relevance <strong>and</strong> coverage
† Interpretation <strong>and</strong> use of the new instrument
4.1. Content relevance <strong>and</strong> coverage
The aims of this validation phase were to determine where evidence for items on the checklist
might be available from assessment opportunities during the programme <strong>and</strong> what opportunities
for collecting evidence were being missed. The first step in this process was to list all the
assessment instruments that were used during the course, when they were used, what function
they performed, what they consisted of, <strong>and</strong> where the results of the assessment were kept for
reference. A table containing the details of 12 instruments can be found in Appendix C.
We distributed this list <strong>and</strong> copies of all the instruments to a seminar of colleagues working in
language testing, <strong>and</strong> asked them to analyse the instruments to determine whether there were
features on the final checklist for which no evidence was being collected during the course <strong>and</strong>
whether there was evidence being collected on the course for which there were no features
mentioned in the checklist.
The main conclusion emerging from the discussion was that there was a great deal of
assessment on the pre-sessional course, both formal <strong>and</strong> informal, but the overall system was not
integrated. Each instrument had its own function <strong>and</strong> its own rationale but the instruments did
not fit together in a coherent whole. The following types of problems were identified:
† Entry on checklist but no evidence available through instruments—for example, the checklist
asks for a judgement concerning the student’s ability to participate in small group
discussions, but there is no structured way of gathering this evidence.
† Evidence available but no entry on checklist—for example, the tutor feedback form for oral
presentations includes several features of oral performance which do not appear on the
checklist.
† Terminology: some of the terminology in the checklist <strong>and</strong> in the instruments was not defined
so it was not clear whether they were referring to the same thing—for example, the TEEP
(Test of English for Educational Purposes) attribute writing scale, which is used to assess the
students’ first piece of writing, contains a criterion called “cohesion”. There is no equivalent
term on the checklist. The nearest notion seems to be “Can make appropriate use of heading
<strong>and</strong> subheadings,” but this may have been quite different from what the TEEP designers
originally had in mind.
† What do these things look like in the classroom? The checklist contained several notions such
as “Can cope with heavy reading load” <strong>and</strong> “Can underst<strong>and</strong> gist” but it was not clear what
the teachers should look for to confirm whether students were able to do these things.
It was to be expected that there would be inconsistencies within the overall system, given that
different instruments <strong>and</strong> practices were designed at different times in the history of the
development of the programme. We have begun to harmonise all of the components so that
evidence is available for all the features specifically mentioned in the checklist.