IELTS Research Reports

An empirical investigation of the process of writing Academic Readingtest items for the International English Language Testing SystemWhile a number of language test development handbooks offer advice on suitable item types fortesting reading and suggest criteria for judging test items (Weir 1993, Alderson 2000, Hughes 2003)the work of the item writer remains under-researched. Studies have been undertaken to investigate thethought processes involved on the part of candidates in responding to IELTS test tasks (Mickan andSlater 2000, Weir et al 2009a and 2009b) and on the part of examiners in scoring IELTS performance(Brown 2003, 2006, Furneaux and Rignall, 2007, O’Sullivan and Rignall 2007), but no research is yetavailable on how IELTS item writers go about constructing test items and translating test specificationsinto test tasks.3 RESEARCH METHODOLOGY AND DESIGN3.1 Deduction and InductionThe review of previous research and current theory and practice related to high-stakes test item-writingunderlines the complexity of the process. Its investigation is likely to involve qualitative as well asquantitative data collection and analyses, inductive as well as deductive approaches. In the analysis of thereading texts selected and adapted by our participants, for example, models already established are useddeductively to produce theory-based quantitative measures of difficulty, word frequency and readability -for example the Academic Word List (AWL) (Coxhead 2000), word frequency levels based on the BritishNational Corpus (BNC) (Cobb, 2003) and indices of readability (Crossley et al 2008).However, for the participant discussions relating to text search, selection, adaptation, item writing anditem editing (audio-recorded with the permission of the participants) a generally inductive approachto data analysis is used. In this process observations are made with the expectation of contributingqualitative insights to a developing theory, seeking processes and patterns that may explain our ‘how’and ‘why’ questions. Patton (1990, p 390) sees such inductive qualitative analysis as permittingpatterns, themes, and categories of analysis to ‘emerge out of the data rather than being imposed onthem prior to data collection and analysis’. Dey (1993, p 99) finds that induction allows a naturalcreation of categories to occur with ‘the process of finding a focus for the analysis, and reading andannotating the data’. As our description of the project’s discussion sessions in Section 6 below willindicate, the analysis ‘moves back and forth between the logical construction and the actual data in asearch for meaningful patterns’ (Patton, 1990, p 411). The meaning of a category is ‘bound up on theone hand with the bits of data to which it is assigned, and on the other hand with the ideas it expresses’(Dey, 1993, p102).3.2 DesignThe research was undertaken in two phases. In the first, an open-ended questionnaire (see Appendix B)was distributed to the item writers accepting our invitation to participate. Questionnaire respondentsincluded all seven Phase 2 participants and three other experienced item writers from the UK,Australia and New Zealand. The instrument elicited data relating to their background and experience,served to contextualise the second, in-depth focus group phase of the study and informed the analysesof the item writer interview and focus group sessions described below.IELTS Research Reports Volume 11277