Assessing and reporting performances on pre-sessional EAP courses

More documents

Recommendations

Info

J. Banerjee, D. Wall / Journal of English for Academic Purposes 5 (2006) 50–69 57 that departments tend to value “communicative effectiveness,” but that this does not necessarily encompass linguistic accuracy. Errey’s introspective verbal report study with lecturers from five different departments at Oxford Brookes University has revealed 24 factors that influence the lecturers’ judgements of student writing. These factors are presented in Table 3. A few researchers (e.g., Geoghegan, 1983; Banerjee, 2003) have explored study needs from the students’ perspective by drawing on their on-going study experiences. Banerjee (2003) conducted a longitudinal study of 25 postgraduate students, interviewing them at a number of different points during their Masters degrees. She found that they experienced difficulties (she termed this “cost”) stemming from factors related to both language <strong>and</strong> the courses they were following (Table 4). She also found that students experienced difficulties in adjusting to the British system of education, particularly the lecture-seminar structure of their courses, the emphasis on learner independence, the dem<strong>and</strong> for critical <strong>and</strong> independent thought, academic conventions, the criteria used for marking (particularly the way they were applied), the role of the teacher, <strong>and</strong> the emphasis in many courses on group work. To summarise, our previous experience of <strong>reporting</strong> student performance on the pre-sessional EAP course <strong>and</strong> the studies reported above narrowed our options with respect to the <strong>reporting</strong> mechanisms that we could consider. It was clear that the use of an external EAP test would risk construct under-representativeness <strong>and</strong> would undermine the usefulness of the report for the admitting departments <strong>and</strong> the students. Our experience of using profile reports based on tutors’ personal criteria was similarly unsatisfactory. We wanted to devise a report form which would incorporate key features from the research we have reviewed <strong>and</strong>, because much of this focused on writing, would also incorporate features from scales such as those found in the English Speaking Union (ESU) framework (Carroll & West, 1989). We would complement this with our own experience, not only as pre-sessional <strong>and</strong> in-sessional tutors but also as members of an academic department. In the sections that follow, we describe both the design of an exit assessment checklist <strong>and</strong> the initial stages in its validation. 3. Developing a final assessment checklist We took a number of decisions in order to ensure that the new exit assessment procedure provided more useful guidance to tutors, students <strong>and</strong> the admissions officers; was more practical; <strong>and</strong> explicitly reflected theories of the EAP construct. These decisions were: (i) Function <strong>and</strong> audience—The main function of the report would be to give the admissions officers an accurate picture of the student’s abilities. Students would receive a copy of the report, <strong>and</strong> it was hoped they would benefit from receiving a frank account of their strengths <strong>and</strong> weaknesses, but we would not edit the reports to soften the blow for students who were not performing adequately. (ii) Coverage—The report form would explicitly reflect current EAP theory. It would specify the skills <strong>and</strong> strategies that had emerged from our analysis of previous research, the report forms prepared in 2001, <strong>and</strong> our previous experience as EAP teachers <strong>and</strong> academic tutors. We decided that it would be inappropriate to comment on attitude, aptitude, motivation, awareness, or any other quality which was a feature of personality rather than of linguistic or EAP ability. (iii) Evidence—We wanted to make it clear that we were only commenting on features that we had evidence for. We also wanted to make clear the limits of our judgements.
58 J. Banerjee, D. Wall / Journal of English for Academic Purposes 5 (2006) 50–69 For instance, we stated explicitly that the tasks that students performed during the presessional course approximated (but could not fully replicate) the dem<strong>and</strong>s students were likely to encounter in their study contexts. We would indicate how students had performed on the tasks set during the course rather than predicting whether they would do well in their future settings. Our intention was to provide our primary users—the admissions personnel, with an evidential basis for deciding whether or not a student was ready to begin studying on a particular programme. This allowed admissions tutors for different degree programmes to interpret the evidence differently, depending on the aspects of EAP proficiency they considered more important for their fields of study. (iv) Format—It was clear that prose reports were difficult for the tutors to write <strong>and</strong> timeconsuming to edit. They were also difficult for the end-users (primarily university admissions personnel) to interpret. One way of eliminating the problems of composing, editing <strong>and</strong> interpreting was to devise a checklist where tutors would only have to place ticks in columns to indicate whether the student had or had not demonstrated certain abilities. The procedure for devising the checklist was as follows: 1. We drew up an exhaustive list of the features of academic reading, writing, listening <strong>and</strong> speaking abilities. 2. We went through an iterative process of combining some items that were similar <strong>and</strong>, in contrast, breaking down some broad descriptors into sub-categories. For instance, we split an item taken from Errey (2000)—“content’”—into two sub-categories, “can analyse the topic of the assignment” <strong>and</strong> “can produce relevant content”. 3. We grouped items according to the language skill they represented. At times these decisions were complex. For example, “providing sufficient evidence” (Ginther & Grant 1996) isa writing skill but is also dependent on the student’s ability to underst<strong>and</strong> <strong>and</strong> make use of reading. We resolved this particular problem by breaking the skill down into two components “can analyse argumentation in academic texts” (classified in the checklist as a reading skill) <strong>and</strong> “can reproduce others’ ideas using his/her own words” (classified as a writing skill). 4. The checklist was formatted to include two columns for each can-do descriptor: “yes” (the student can do this) or “must pay attention to” (this skill or strategy). 5. The draft checklist was then presented to the University’s postgraduate admissions officer <strong>and</strong> to the pre-sessional course tutors. Changes were made on the basis of their feedback. 6. The revised checklist was piloted by the pre-sessional course tutors on three of their students (a strong student, a weak student, <strong>and</strong> one whose ability fell between these two extremes)—approximately 27 students. On the basis of this pilot procedure, further changes were made. 7. Taking account of recommendations by Weigle (1994) for rater familiarisation with the adopted rating scale, there was extensive opportunity, at each point during the consultation <strong>and</strong> piloting phase, for discussion of what the can-do descriptors meant <strong>and</strong> how they might be applied. Indeed, the raters (the pre-sessional course tutors) contributed to the content, wording <strong>and</strong> final format of the exit assessment checklist. The details of this consultation are presented in Section 3.1 (below). 8. We also met with the students in order to introduce the checklist to them as well as to elicit their feedback on the items included. Changes were made on the basis of this feedback.
Page 1 and 2: Assessing
Page 3 and 4: 52 J. Banerjee, D. Wall / Journal o
Page 7: 56 J. Banerjee, D. Wall / Journal o
Page 17 and 18: 66 Can take satisfactory notes Vari
Page 19 and 20: 68 Instrument When used Function De

Assessing and reporting performances on pre-sessional EAP courses

Create successful ePaper yourself

Delete template?

Save as template?