IELTS Research Reports

An empirical investigation of the process of writing Academic Readingtest items for the International English Language Testing System4.2.2 Participant focus group discussionsThe experienced group participated next in a focus group discussion structured around a set of ninesemantic differential continua (Osgood, 1957, using the unlabelled scale format compared with otherformats by Garland, 1996) and as seen in Table 8. In the table, summaries of the comments made bythe participants in their 20 minutes of unmediated discussion are placed in their approximate locationon the continua for the nine scales. The adjectives for the continua were selected by the researchers.Again, points made by participants in the focus group discussion served to triangulate views expressedin the preceding interview activity concerning IELTS text search and treatment and item development:flowcharts and discussions already reported. Following discussion of the semantic differentials, theresearch team pursued emerging issues with the group.The experienced group, like the non-experienced, expressed uncertainty about candidates’ level ofEnglish language proficiency. The four discussed the need to keep the candidates in mind when writingitems, but agreed that it was challenging to do this, given the ‘the variety of the situation and [thecandidates’] levels of English’. Each participant had their own points of reference for these. Anne alsoworked as an examiner for the speaking paper and so met many candidates while both William andElizabeth had experience of preparing students for the test. However, Elizabeth reminded the groupthat the candidates they met in the UK would not be representative of the full range of candidatestaking the test - especially those from relatively underprivileged backgrounds.Item writers also received information about candidates from IELTS. An annual report on demographicdata is provided by Cambridge ESOL and ‘common wrong answers’ to open response items arediscussed at pretest review meetings. What Anne described as the ‘off the wall’ nature of some ofthese wrong answers and the observation that ‘some people have been accepted at universities, where Ithought their English was totally inadequate’ led William to the conclusion that ‘you can do reasonablywell on IELTS, I think. And still have what seems to be a low level of English’. Elizabeth alsoquestioned whether IELTS candidates would need to arrive at a full understanding of the text in orderto succeed on the questions, suspecting that in IELTS ‘half the time the candidates don’t read the textfrom beginning to end because they don’t have to’ because local details in the text were being testedby the items rather than the overall meaning. However, Anne wondered whether William’s concerncould be justified as success on the test would require adequate levels of performance on the directspeaking and writing papers as well as reading and listening.There was discussion of how the participants had developed their item writing expertise. For Jane thiswas not easy to explain: ‘It’s difficult to say sometimes exactly what you’re doing and how you’redoing it’. Anne agreed, observing that ‘the processes you go through aren’t necessarily conscious’.However, there were item writing skills that could be learnt. Anne had come to appreciate theimportance of ‘working the task’: attempting it as a candidate would. Jane agreed that this was helpful,but admitted she rarely did this prior to submission because of the pressure of deadlines. Elizabeth hadfound very helpful the advice given to her at her initial training session to focus on what she felt to bethe key points of the text, finding that this could help her when she was ‘stuck on something’.Anne felt that her items had improved ‘over years of seeing other peoples’ and having to mend yourown’. William pointed to the value of attending editing meetings to obtain insights and Elizabeth feltthat feedback at editing meetings had been one of her main sources of learning about item writingespecially where a the chair of the meeting, as an experienced and successful item writer, had beeneffective at showing how a text or item could be improved.IELTS Research Reports Volume 11297