IELTS Research Reports

An empirical investigation of the process of writing Academic Readingtest items for the International English Language Testing SystemThe trained writers seemed more ready to edit their texts; reshaping them if necessary to meet therequirements of the items. Of the untrained writers Mary seemed to have the strongest objections torevising her text, but in fact made the most substantial changes of this group. These changes includedmoving material between paragraphs to square her text with the items she wanted to use.In sum, the effect of editing for both groups, apparent in the analysis of the submitted texts and fromthe discussions in the editing meetings, was to increase the coherence and information density ofthe texts and to make them more accessible to readers from non-English speaking backgrounds. Thechanges also served to reduce technical and cultural specificity, colloquialism, journalistic touches(such as sensationalism, personal engagement of writer etc.) and, particularly in the case of theexperienced group’s texts, to reduce the repetition of ideas.In devising their items, both groups made use of a range of item types. The True/ False/ Not Given(T/F/NG) item type was chosen most often across groups, but no clear differences in item typeselection could be seen from the small sample submitted.As was to be expected, the experienced item writers submitted items of better quality and were betterable to correct the problems that they found. A greater number of shortcomings that would breachthe IELTS item writer guidelines could be identified in the untrained writers’ submissions. For someuntrained writers, items within sets did not consistently follow the order of information in the textwhere this would usually be expected (as in Mary’s MCQ items: 15 and 17 concern the first paragraph,16 is associated with the sixth paragraph and the necessary information for item 18 is distributedthroughout the text). Items within a set were sometimes isolated from each other: Mathilda’s item 17,for example, relates to her ninth paragraph while the rest of her T/F/NG items are associated with thefirst four paragraphs of her text.The items submitted by the untrained writers sometimes addressed the same parts of the text morethan once. Victoria, for example, has three pairs of items that seem to address the same sentences inher text (items 2 and 3; 4 and 5; and 8 and 13). Untrained item writers’ texts included stretches ofuntested material: five of Victoria’s 16 paragraphs did not include information required to respond toany of her items.The non-experienced writers felt that their lack of guidance about the test inhibited their ability toproduce adequate items. They felt that they would have benefited from information on devising MCQdistractors and on the skills being targeted by items of different types. It should be noted that thesewriters had been directed to the Teaching Resources section of the IELTS website, which providessome guidance on this question under the heading of ‘What skills are tested in this task type?’However, the information is inexplicit. For Task Type 8 - Identification of Writer’s Views/Claims orof Information in a Text, the explanation is as follows:The first variation of this task type aims to test the candidate’s ability to recognise opinions or ideas,and is thus often used with discursive or argumentative texts.This is not clear enough to guide an item writer. The intended relationship between the items andthe text is not made plain and so the type of reading required is not explicit. The lack of guidance isreflected in the very different ways in which Mathilda and Victoria interpreted this task type.In the editing meeting, the non-experienced group were relatively less critical of each others’ work(although it should also be noted that, unlike the experienced group, they had not met each otherbefore the day of the editing meeting). The experienced writers appeared far more efficient in theirIELTS Research Reports Volume 11323