IELTS Research Reports

Anthony Green and Roger HawkeyWilliam spoke of having learnt how to devise plausible distractors for multiple choice items. However,there were limits to how far this could be learnt as an item writing skill and he wondered about the roleof background knowledge in eliminating incorrect options: ‘I think there’s a risk with IELTS becauseif it’s a scientific text, I may not know nearly enough to know what would be a plausible distractor.What seems plausible to me could be instantly rejected by somebody who knows a little more aboutthe subject.’Testing implicit information was seen to be problematic. There were cases of disagreement betweenthe item writers and their colleagues carrying out pre-editing reviews about ‘whether [a point] isimplicit, but strongly enough there to be tested or not’ (William). For Jane, testing the writer’sinterpretation against others’ was a further argument in favour of the pre-editing and editing processes:‘fresh eyes are invaluable when it comes to evaluating a task’.Although Jane reported that she tried to keep the level of language in mind as she wrote, the groupagreed that the difficulty of items was not easy to predict. None of the writers seemed to have a clearsense of the proportion of items associated with a text that a successful IELTS candidate at band 6.0 or6.5 might be expected to answer correctly. Pretesting results often revealed items to be easier or moredifficult than expected.5 ANALYSIS AND FINDINGS ON THE TEXTSThe analysis here is applied to the texts as they were submitted by the seven participants, before anychanges made during the public editing process reported below. The texts and items submitted by theitem writers (in their adapted, but unedited state) are presented in Appendix C. This analysis showshow the texts were shaped by the writers and so serves to contextualise the comments made in theinterview and focus group sessions.In this section, we again begin with the texts submitted by the non-experienced group. FollowingWeir et al. (2009a) we employed automated indices of word frequency and readability to inform andsupplement our qualitative text analyses. Outcomes of these procedures are given in Figures 1 to 3below and are discussed in relation to each submission in the following section.298 www.ielts.org