IELTS Research Reports

Anthony Green and Roger HawkeyShe found that in comparison to novices, more expert item writers, those producing more positivelyevaluated texts and items that met the requirements of the test developers (UK examining boardsoffering tests of English as a Foreign Language),■■■■■■■■■■■■are more aware of the test specifications and are quickly able to recognise texts that showpotential as test material. Where novices tended to devise a listening script from a sourcetext first and then to write the questions, experts were more inclined to start from thequestions and then to build a script to fit with these.are more aware of the needs of candidates for clear contextual information and are betterable to provide accessible contextualising information in the form of short, accessiblerubrics and co-text.explore a range of possible task ideas rather than committing immediately to one thatmight later prove to unworkable.use many more learned rules or ruses than non-experts including, for example,exchanging words in the text and in the question so that the hypernym appears in the textadding additional text to the script to introduce distraction and reduce the susceptibility ofthe questions to guessing strategiesAlthough more experienced item writers tended to outperform the recently trained, expertise wasnot simply a function of experience. One writer with no previous experience of test item writingperformed better in the judgement of a review panel than two item writers with extensive experience(Salisbury 2005). Salisbury also concludes that expertise in listening test item writing is collective innature. Individual writers rarely have sufficient capability to meet institutional requirements at the firstattempt and need the feedback they receive from their colleagues to achieve a successful outcome. Itmight be added that item writer expertise itself is not sufficient to guarantee test quality. Even whereitems are subject to rigorous review, piloting usually reveals further deficiencies of measurement.The Cambridge ESOL approach to test development is described in detail by Saville (2003) and byKhalifa and Weir (2009). The IELTS test production process for the reading and listening papersis outlined in a document available from the IELTS website, www.ielts.org. The goal of this testproduction process is that ‘each test [will be] suitable for the test purpose in terms of topics, focus,level of language, length, style and technical measurement properties’ (IELTS 2007, 1).IELTS test material is written by freelance item writers externally commissioned by CambridgeESOL in a process centrally managed from Cambridge and carried out according to confidentialtest specifications or item writer guidelines laid down by the test developers (although see Clapham1996a, 1996b for an account of the role of externally commissioned item writing teams in developingthe IELTS academic reading module). These guidelines, periodically modified to reflect feedbackfrom item writers and other stakeholders, detail the characteristics of the IELTS modules (speaking,listening and academic or general training reading and writing), set out the requirements forcommissions and guide writers in how to approach the item writing process. The guidelines cover thesteps of selecting appropriate material, developing suitable items and submitting material. However,a good deal of the responsibility for test content is devolved to the externally commissioned workersincluding the item writers and their team leaders or chairs for each of the modules. Khalifa and Weir(2009) describe the chair as having responsibility for the technical aspects of item writing and forensuring that item writers on their team are fully equipped to generate material of the highest quality.According to the Cambridge ESOL website (Cambridge ESOL n.d.) the overall network of Cambridgeitem writers working across the Cambridge ESOL product range includes 30 chairs and 115 item274 www.ielts.org