Assessing and reporting performances on pre-sessional EAP courses

More documents

Recommendations

Info

J. Banerjee, D. Wall / Journal of English for Academic Purposes 5 (2006) 50–69 63 student self-assessment. Since, the exit assessment report represented high stakes for the students they might well have described themselves in a way that they thought appropriate in order to successfully complete the course. However, the tutors we interviewed felt that their students had largely been forthright in their self-assessments <strong>and</strong> individual tutorials. In fact, one tutor commented that the students “were more honest than perhaps we give them credit for.” Indeed many students encouraged their tutors to be “truthful” so that they could set learning goals for the rest of the year. Nevertheless, a number of issues also emerged from the interviews. Most of them are clearly training issues which need to be addressed as soon as possible: 1. The checklist is criterion-referenced <strong>and</strong> each student is to be judged against the can-do statements. However, at least one tutor not only considered whether the student had the stated ability or had had difficulty but also tried to rank his students. For example, in the case of the writing item “Can produce grammatically correct text” he ticked that a student had this ability not because she performed well in this area but because he wanted to distinguish her from another good student who did not. 2. During the consultation <strong>and</strong> piloting phase (see Section 3.1, above), the tutors specifically requested a column entitled “we have no evidence” to cover cases where students did not demonstrate a particular ability. The purpose of this column was discussed extensively at the time yet, during the implementation of the checklist, at least one tutor agonised over using the “no evidence” column, worrying that it would imply that the student did not have that language ability. 3. The exit assessment checklist includes a summary judgement on the final page. This was included at the request of the course tutors (see Section 3.1, above). However, the interviews revealed that the route by which the tutors arrived at their summary judgements differed. One tutor said that she had a global sense of whether a student had made satisfactory progress or not <strong>and</strong> based her judgement on that rather than on the distribution of ticks across the report. Another tutor based her judgement on the number of ticks the student received in the “has had some difficulties” column as well as her judgement of the gravity of these ticks. This last practice is particularly interesting for it introduces a level of detail not explicit in the report form. This has implications for the relationship between the detailed checklist <strong>and</strong> the summary section of the report form <strong>and</strong> is again an issue to be dealt with in training. Though there is still much to be done to help tutors adjust to the new <strong>reporting</strong> system <strong>and</strong> to orientate new users of the checklist, the remaining concern relates to how the tutors applied the “can-do” statements. It is important to remember that the tutors had discussed their interpretations of items during the consultation <strong>and</strong> piloting stage (described earlier) <strong>and</strong> were therefore familiar with the checklist. Indeed, they had contributed to its wording. Yet, as Lumley (2002) points out, when making their judgements all raters have to reconcile the rating scale (in this case the can-do statements) <strong>and</strong> their observation <strong>and</strong> interpretation of student <strong>performances</strong>. This proved to be an issue for the tutors we interviewed. For instance, one tutor commented on the word “satisfactory” in the statement “can take satisfactory notes.” Since, notes are generally for personal use, this tutor wondered whether she was the best judge of whether a student’s notes were “satisfactory” for the student’s own purposes. Much can be done to discuss with the course tutors the behaviours that constitute evidence of a particular ability (as well as the behaviours that suggest difficulty). However, the tutor is still the ultimate arbitrator <strong>and</strong>, as Lumley (2002) argues, it is not possible to cover all eventualities during rater training.
64 J. Banerjee, D. Wall / Journal of English for Academic Purposes 5 (2006) 50–69 5. Planning future directions In this paper, we have presented the rationale for a new assessment instrument <strong>and</strong> the process we went through to tailor the instrument to our needs <strong>and</strong> to investigate its validity. Our work so far has identified two areas for immediate action: 1. Integrating continuous assessment with final assessment—Our review of the instruments we use for continuous assessment (Appendix C) clearly shows that there is a need to review <strong>and</strong> rationalise our system, so that each formative judgement contributes to the summative checklist. This should also provide our students with sufficient opportunities to demonstrate whether they have mastered the abilities in the final checklist. This will, of course, have an impact upon syllabus design. 2. Training the tutors—Despite our involvement of course tutors in the development of the checklist, it is clear that they still diverged in their interpretation <strong>and</strong> implementation of the instrument. This problem could be exacerbated in cases where tutors are teaching on the course for the first time. We must prepare materials <strong>and</strong> procedures that we can use with these tutors, both in our orientation work before teaching begins <strong>and</strong> during the rest of the course, to make sure that they underst<strong>and</strong> the different meanings behind terms such as ‘coherence’ <strong>and</strong> to bring all tutor judgements into line. Our investigations so far have identified key elements to be included in the training material. This study has also revealed a number of longer-term objectives. We are keenly aware that we are still in the early stages of design <strong>and</strong> that there is work to be done before we can be satisfied that this form of assessment is entirely appropriate for our situation. There are two further challenges which must be dealt with in the near future: 1. Interpretation <strong>and</strong> use of the instrument by admissions personnel—One of our aims for this checklist was to disambiguate the information that the pre-sessional exit assessment procedure provides to admitting departments. The preliminary feedback that we received (see Section 3.1, above) indicated that we had succeeded in this aim. However, a further phase of validating the checklist should explore the ways in which admissions personnel interpret the results of the checklist <strong>and</strong> arrive at a decision on whether or not to admit a conditional applicant. 2. Refining the construct—The checklist that we have designed was based on our underst<strong>and</strong>ing of the issues present in the literature on EAP assessment, our own experience both as EAP tutors <strong>and</strong> as academic lecturers, <strong>and</strong> the experience of colleagues in other universities in Britain <strong>and</strong> other parts of the world. Despite the rich resources available, there is still much to learn about the dem<strong>and</strong>s placed on students at different levels in the university system (undergraduates <strong>and</strong> postgraduates) <strong>and</strong> in all the different departments that our students will be entering. In particular, we need to underst<strong>and</strong> better what aspects of language ability are crucial in determining students’ success in meeting these dem<strong>and</strong>s. Investigating this will involve in-depth interviews with academic staff <strong>and</strong> both native <strong>and</strong> non-native English speaking students, bearing in mind that students have been found to construct ‘success’ rather differently from their tutors (Basturkmen & Lewis, 2002). The issues involved in carrying out end-of-course assessment on pre-sessional courses have not been well covered in the literature, yet the assessment that takes place on these courses is high-stakes for the students involved. Failure to meet the st<strong>and</strong>ards set could result in the students being refused entry to the university of their choice. Our survey of assessment practices on pre-sessional courses has revealed that EAP professionals are aware of these pressures <strong>and</strong>
Page 1 and 2: Assessing
Page 3 and 4: 52 J. Banerjee, D. Wall / Journal o
Page 13: 62 J. Banerjee, D. Wall / Journal o
Page 17 and 18: 66 Can take satisfactory notes Vari
Page 19 and 20: 68 Instrument When used Function De

Assessing and reporting performances on pre-sessional EAP courses

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?