IELTS Research Reports

Anthony Green and Roger Hawkey1 AIMSThis research report describes a study of reading, test text selection, item writing and editing processes,areas of test production that have rarely been transparent to those outside testing organisations. Basedon retrospective reports, direct observation and analyses of the texts produced, the report compares howtrained and untrained item writers select and edit reading texts to make them suitable for a task-basedtest of reading and how they generate the accompanying items. Both individual and collective editingprocesses are investigated. The analyses in the study are expected to inform future high-stakes readingtest setting and assessment procedures, in particular for examination providers.2 BACKGROUND AND RELATED RESEARCH2.1 A socio-cognitive test validation frameworkThe research is informed by the socio-cognitive test validation framework (Weir 2005), which underpinstest design at Cambridge ESOL (Khalifa and ffrench, 2008). The framework, further developed at theCentre for Research in Language Learning and Assessment (CRELLA) at the University of Bedfordshire,is so named because it gives attention both to context and to cognition in relating language test tasksto the target language use domain. As outlined in Khalifa and Weir (2009) and Weir et al (2009a and2009b), in the socio-cognitive approach difficulty in reading is seen to be a function of i) the complexityof text and ii) the level of processing required to fulfil the reading purpose.In Weir et al (2009a) IELTS texts were analysed against twelve criteria derived from the L2 readingcomprehension literature (Freedle and Kostin 1993, Bachman et al. 1995, Fortus et al. 1998, Enrightet al. 2000, Alderson et al., 2004 and Khalifa and Weir 2009a) These criteria included: Vocabulary,Grammar, Readability, Cohesion, Rhetorical organisation, Genre, Rhetorical task, Pattern ofexposition, Subject area, Subject specificity, Cultural specificity and Text abstractness. In the currentstudy, we again employ such criteria to consider the texts produced by item writers and to analyse thedecisions they made in shaping their texts.In Weir et al (2009b) the cognitive processes employed by text takers in responding to IELTS readingtasks are analysed, with a particular focus on how test takers might select between expeditious andcareful reading and between local and global reading in tackling test tasks.Local reading involves decoding (word recognition, lexical access and syntactic parsing) andestablishing explicit propositional meaning at the phrase, clause and sentence levels while globalreading involves the identification of the main idea(s) in a text through reconstruction of its macrostructurein the mind of the reader.Careful reading involves extracting complete meanings from text, whether at the local or global level.This is based on slow, deliberate, incremental reading for comprehension. Expeditious reading, incontrast, involves quick, selective and efficient reading to access relevant information in a text.The current study was expected to throw light on how the item writers might take account of theprocesses engaged by the reader/ test taker in responding to the test tasks and how item writers’conceptions of these processes might relate to reading for academic study.272 www.ielts.org