Assessing and reporting performances on pre-sessional EAP courses
Assessing and reporting performances on pre-sessional EAP courses
Assessing and reporting performances on pre-sessional EAP courses
Create successful ePaper yourself
Turn your PDF publications into a flip-book with our unique Google optimized e-Paper software.
J. Banerjee, D. Wall / Journal of English for Academic Purposes 5 (2006) 50–69 61<br />
stage of developing <str<strong>on</strong>g>and</str<strong>on</strong>g> reviewing this new assessment format, we investigated the validity of<br />
the exit assessment in two phases:<br />
† C<strong>on</strong>tent relevance <str<strong>on</strong>g>and</str<strong>on</strong>g> coverage<br />
† Inter<strong>pre</strong>tati<strong>on</strong> <str<strong>on</strong>g>and</str<strong>on</strong>g> use of the new instrument<br />
4.1. C<strong>on</strong>tent relevance <str<strong>on</strong>g>and</str<strong>on</strong>g> coverage<br />
The aims of this validati<strong>on</strong> phase were to determine where evidence for items <strong>on</strong> the checklist<br />
might be available from assessment opportunities during the programme <str<strong>on</strong>g>and</str<strong>on</strong>g> what opportunities<br />
for collecting evidence were being missed. The first step in this process was to list all the<br />
assessment instruments that were used during the course, when they were used, what functi<strong>on</strong><br />
they performed, what they c<strong>on</strong>sisted of, <str<strong>on</strong>g>and</str<strong>on</strong>g> where the results of the assessment were kept for<br />
reference. A table c<strong>on</strong>taining the details of 12 instruments can be found in Appendix C.<br />
We distributed this list <str<strong>on</strong>g>and</str<strong>on</strong>g> copies of all the instruments to a seminar of colleagues working in<br />
language testing, <str<strong>on</strong>g>and</str<strong>on</strong>g> asked them to analyse the instruments to determine whether there were<br />
features <strong>on</strong> the final checklist for which no evidence was being collected during the course <str<strong>on</strong>g>and</str<strong>on</strong>g><br />
whether there was evidence being collected <strong>on</strong> the course for which there were no features<br />
menti<strong>on</strong>ed in the checklist.<br />
The main c<strong>on</strong>clusi<strong>on</strong> emerging from the discussi<strong>on</strong> was that there was a great deal of<br />
assessment <strong>on</strong> the <strong>pre</strong>-sessi<strong>on</strong>al course, both formal <str<strong>on</strong>g>and</str<strong>on</strong>g> informal, but the overall system was not<br />
integrated. Each instrument had its own functi<strong>on</strong> <str<strong>on</strong>g>and</str<strong>on</strong>g> its own rati<strong>on</strong>ale but the instruments did<br />
not fit together in a coherent whole. The following types of problems were identified:<br />
† Entry <strong>on</strong> checklist but no evidence available through instruments—for example, the checklist<br />
asks for a judgement c<strong>on</strong>cerning the student’s ability to participate in small group<br />
discussi<strong>on</strong>s, but there is no structured way of gathering this evidence.<br />
† Evidence available but no entry <strong>on</strong> checklist—for example, the tutor feedback form for oral<br />
<strong>pre</strong>sentati<strong>on</strong>s includes several features of oral performance which do not appear <strong>on</strong> the<br />
checklist.<br />
† Terminology: some of the terminology in the checklist <str<strong>on</strong>g>and</str<strong>on</strong>g> in the instruments was not defined<br />
so it was not clear whether they were referring to the same thing—for example, the TEEP<br />
(Test of English for Educati<strong>on</strong>al Purposes) attribute writing scale, which is used to assess the<br />
students’ first piece of writing, c<strong>on</strong>tains a criteri<strong>on</strong> called “cohesi<strong>on</strong>”. There is no equivalent<br />
term <strong>on</strong> the checklist. The nearest noti<strong>on</strong> seems to be “Can make appropriate use of heading<br />
<str<strong>on</strong>g>and</str<strong>on</strong>g> subheadings,” but this may have been quite different from what the TEEP designers<br />
originally had in mind.<br />
† What do these things look like in the classroom? The checklist c<strong>on</strong>tained several noti<strong>on</strong>s such<br />
as “Can cope with heavy reading load” <str<strong>on</strong>g>and</str<strong>on</strong>g> “Can underst<str<strong>on</strong>g>and</str<strong>on</strong>g> gist” but it was not clear what<br />
the teachers should look for to c<strong>on</strong>firm whether students were able to do these things.<br />
It was to be expected that there would be inc<strong>on</strong>sistencies within the overall system, given that<br />
different instruments <str<strong>on</strong>g>and</str<strong>on</strong>g> practices were designed at different times in the history of the<br />
development of the programme. We have begun to harm<strong>on</strong>ise all of the comp<strong>on</strong>ents so that<br />
evidence is available for all the features specifically menti<strong>on</strong>ed in the checklist.