Assessing and reporting performances on pre-sessional EAP courses

More documents

Recommendations

Info

J. Banerjee, D. Wall / Journal of English for Academic Purposes 5 (2006) 50–69 59 9. The final form of the exit assessment checklist (Appendix B) was used during the last week of the pre-sessional course. Each student received their report <strong>and</strong> a copy was sent to the University’s postgraduate admissions officer <strong>and</strong> the admissions personnel in admitting departments. 3.1. Consulting the stakeholders <strong>and</strong> piloting the checklist As has already been stated, the aims of the consultation <strong>and</strong> piloting phase were threefold: † To provide some of our stakeholders (the University’s admissions personnel <strong>and</strong> the presessional course tutors) with an opportunity to comment on the initial draft <strong>and</strong> suggest changes or additions before the piloting stage (we consulted students during the piloting stage). † To pilot the revised instrument in order to ascertain its usability <strong>and</strong> coverage. † To maximise the course tutors’ opportunities to familiarise themselves with the checklist <strong>and</strong> to arrive at shared interpretations of each of the ‘can-do’ statements. During the consultation <strong>and</strong> piloting phase, we received valuable feedback in a number of areas: (i) Usability of the report form—The tutors found the report form quick to complete <strong>and</strong>, despite some suggestions for changes, they believed that they understood the can-do statements. The University’s Postgraduate Admissions Officer noted that the form provided judgements in four skill areas, commenting that this paralleled the report formats of language proficiency test scores like the IELTS. She thought this was a step forward, arguing that this form was more transparent than its predecessor <strong>and</strong> that the information was easy to retrieve because it was not presented in prose. She believed that course directors in the academic departments would be able to return to this form at a later date in order to compare the students’ on-going performance in their departments with what was reported at the end of the EAP course. (ii) Format of the overall report—The tutors initially argued that if a student seemed to be satisfactory in all respects it would be sufficient to present a summary judgement of his/her performance on the course. They suggested that the form for such students should consist of a single page with a box to tick for each of the four skills (listening, speaking, reading <strong>and</strong> writing). We resisted this suggestion for two reasons. The first was that we needed to be sure (inasmuch as this is possible) that all the tutors shared the same underst<strong>and</strong>ing of the construct for each skill area. We felt that if they had to place a tick opposite each aspect listed under each skill they would be obliged to consider all the traits that we considered to be important. The second reason had to do with accountability <strong>and</strong>, ultimately, face validity: we felt it unlikely that the University would accept our judgements on the adequacy or otherwise of students’ <strong>performances</strong> unless we demonstrated how we had arrived at our decisions. Nevertheless, the course tutors prevailed upon us to include a summary judgement page (this became the final page of the exit assessment form). They argued that they would feel more comfortable being “honest” in the checklist if they could also
60 J. Banerjee, D. Wall / Journal of English for Academic Purposes 5 (2006) 50–69 indicate how seriously they viewed a student’s particular difficulties. They would not worry about ticking individual boxes if they could indicate in the summary section that they were “generally satisfied with the student’s performance”. This would allow them to give their students specific feedback <strong>and</strong> guidance without suggesting that they had not “successfully completed” the course. (iii) Wording of the judgement—The tutors argued that some of the diplomacy of the original <strong>reporting</strong> format had been lost <strong>and</strong> that, in order to correct the apparent baldness of the claims being made, the descriptors needed to be presented within the context in which the judgements took place. The outcome of this discussion was that all the judgements are prefaced with the phrase “Our evidence suggests that.” The tutors also argued that it should be possible to indicate when there was no evidence for a particular judgement. For instance, a skill may have been taught but the student may not have taken advantage of opportunities to demonstrate it. This resulted in the inclusion of the third column, “We have no evidence.” It should be noted at this point that the tutors’ request for this column proved somewhat problematic when the report was used in its final form (see below). Thirdly, in line with the spirit of the new exit assessment, the tutors rejected the formulations “will have difficulty” <strong>and</strong> even “has difficulty” as they implied a prediction of future performance. They preferred instead to locate their judgements within the period of the course, hence the formulation “has had difficulty”. (iv) Evidence available on which to base a judgement—Though the tutors considered the report form easy to use <strong>and</strong> transparent, they were concerned about the extent to which they could make detailed judgements about the students’ listening <strong>and</strong> reading abilities. They argued that since the processes involved in listening <strong>and</strong> reading are internal, they would be reliant on student self-report (which might not be reliable) or on their interpretation of students’ listening <strong>and</strong> reading abilities based on what they produced when speaking <strong>and</strong> writing. This problem has not been fully or satisfactorily resolved. Indeed, it remains one of the difficulties in any assessment of receptive skills. The tutors also argued that students with good language skills might fail to give evidence of certain can-do statements such as strategies like “asking for clarification,” since they might never be in a position where they have misunderstood <strong>and</strong> have to ask for clarification. The solution we have adopted has been to orientate the students to the criteria by which they are to be judged. When the students register they receive a self-assessment checklist which requires them to make judgements about their language abilities at the start of the course. This checklist replicates the can-do statements used in the exit assessment checklist <strong>and</strong> has proved a useful consciousness-raising tool particularly since students discuss their self-assessments during their first consultation with their tutor. Nevertheless, we are still concerned that students might fail to fully demonstrate their abilities according to the “can-do” statements <strong>and</strong> we plan to explore this area in a later study. 4. Investigating the validity of the checklist The final version of the checklist incorporated as much of the feedback as was feasible from the course tutors <strong>and</strong> the University’s Postgraduate Admissions Officer. It was then used to assess 86 students, all of whom completed the 4-week pre-sessional EAP course. In a further
Page 1 and 2: Assessing
Page 3 and 4: 52 J. Banerjee, D. Wall / Journal o
Page 9: 58 J. Banerjee, D. Wall / Journal o
Page 17 and 18: 66 Can take satisfactory notes Vari
Page 19 and 20: 68 Instrument When used Function De

Assessing and reporting performances on pre-sessional EAP courses

Create successful ePaper yourself

Delete template?

Save as template?