Calibrating tests to the CEFR

The European Association for Quality Language ServicesCalibrating tests to the CEFRA simplified guide for EAQUALS Members oncalibrating entry or progress tests to the CEFR© EAQUALS: The European Association for Quality Language Services:Email: info@eaquals.org Internet: www.eaquals.org

The European Association for Quality Language ServicesContentsIntroduction Page 1External Validation Page 2Summary Chart Page 3The Step-by-step Procedure Page 4© EAQUALS: The European Association for Quality Language Services:EAQUALS SECRETARIAT: P.O. Box 95, Budapest, H-1301 HungaryEmail: info@eaquals.org Internet: www.eaquals.orgRegistered Office: Via Torrebianca 18 34132 Trieste, Italy2

The European Association for Quality Language ServicesIntroductionThis guide provides simple procedures to calibrate a placement or progress test to the CEFR.It is not a guide for developing examinations or for relating the results from existing tests orexaminations to the CEFR by determining “cut-scores” that define a minimum performance for aparticular level. A manual is provided by the Council of Europe for that purpose 1 .In order to give information about a learner’s level of CEFR proficiency the test needs to:• be developed to a detailed, balanced specification that relates to the CEFR• have a broad coverage (no the grammar and vocabulary done this month bur the grammarand vocabulary most important at, for example, Level B1);• be appropriate to the context and type of learners concernedIt is perfectly legitimate to calibrate an existing test to the CEFR levels. This guide gives asimple way to externally validate the cut-off scores on the test for each CEFR level throughcorrelating and referencing results to an anchor test already calibrated to the CEFR and/or toteacher judgements with criteria.1 Council of Europe (2009) “Relating Language Examinations to the Common European Framework of Reference for Languages:Learning, Teaching, Assessment (CEFR): A Manual.” Available on www.coe.int/lang.North, B. and Jones J. (2009) “Relating Language Examinations to the Common European Framework of Reference for Languages:Learning, Teaching, Assessment (CEFR): Further material on standard setting through scaling and teacher judgement.” Availableon www.coe.int/lang.© EAQUALS: The European Association for Quality Language Services:1EAQUALS SECRETARIAT: P.O. Box 95, Budapest, H-1301 HungaryEmail: info@eaquals.org Internet: www.eaquals.orgRegistered Office: Via Torrebianca 18 34132 Trieste, Italy

The European Association for Quality Language ServicesExternal ValidationAlthough empirical validation is mainly something examination providers are concerned with,schools can also apply basic principles in a simple way. Data analysis for external validation canbe carried out satisfactorily with nothing more elaborate than the simple correlation functions inMicrosoft Excel and a couple of Microsoft Word tables. Examination providers will no doubt wishto go into more detail and the CEFR Manual written for them also provides a ReferenceSupplement should they need advice on how to do so.External validation is the independent corroboration through the use of an external criterion ofthe claim that a test or assessment procedure reports in terms of CEFR levels. The mainproblem with undertaking such a study is collecting data. All learners operating as subjectsneed to take both the assessment procedure under study and the assessment used as anexternal criterion. Examination providers will want to conduct studies with large populationstudies, but in a school, a sample of 50 learners would be more than sufficient (30 is anabsolute minimum).The technique can be used either to plot entry, progress or exit test scores against an externalcriterion. The test concern:(a) a particular skill, e.g. “Your linguistic knowledge is Level B2”(b) a global result e.g.: “You are Level B2.” This might be the result of averaging results fromsub tests.There are four alternatives for the external criterion:(a) Results in examinations calibrated to CEFR or on a test previously calibrated to the CEFR. Itcould be results from a suite of successive ALTE examinations situated at different levels. Ifthe assessment under study tests a particular skill or aspect, then the test (or suite of tests)used as an external criterion will have to test that skill too; if the assessment under study isa global one, then the external criterion could be the global result in CEFR terms from anexamination or suite of examinations (e.g. Cambridge exams).(b) Judgements made by teachers for the skill concerned could be used, provided they knowthe learners’ work well and that they have been systematically trained in a standardisedinterpretation of the CEFR levels (Activities 4, 5 & 6).(c) More formal school assessments of speaking and writing in relation to the CEFR criteria forstandardisation training in the appendices to the EAQUALS Standardisation Pack. Thesecould be averaged to give a “global” result. Results will be better if two teachers, each ofwhom has attended standardisation staff seminars act as the raters, and if they have tonegotiate a final grade.(d) Self assessments could also be used. Again, self-judgements will be more accurate if thelearners have been trained in relation to the CEFR levels – perhaps in comparing their ownlevel to performances in standardisation training videos.The best thing to do is to collect data from as many sources as possible – and hope one ofthem gives a good correlation.© EAQUALS: The European Association for Quality Language Services:EAQUALS SECRETARIAT: P.O. Box 95, Budapest, H-1301 HungaryEmail: info@eaquals.org Internet: www.eaquals.orgRegistered Office: Via Torrebianca 18 34132 Trieste, Italy2

The European Association for Quality Language ServicesSummary ChartActivity:Validating CEFR Levels reported by Entry, Progress or Exit TestsMaterials: o Exam results across a range of CEFR levels from a suite of examinationslinked to the CEFR (e.g. Cambridge) or from a test reporting across a numberof CEFR levelso CEFR Assessment procedures implemented for speaking and writing afterStandardisation training as in Activities 2-4 (Speaking) and 9 (Writing) above.o Data for 50 or more learners, with as wide a range of level as possible: TEST score for each student on the local test EXAM result for each student on the examination or suite of examinationsconcerned ASSESSMENTS results for each student from assessment of speaking (withCEFR Table 3 criteria grid) and writing (with CEFR writing criteria grid) SELF-assessed CEFR level from CEFR Global self-assessment scale orPortfolio self-assessment grid TEACHER informal assessment with CEFR scales and/or Portfolio checklists(Definitely not advisable on extensive courses with large classes)o Microsoft Excel & WordSteps:1. Prepare the data.a. Convert exam results to numbers; Example: with Cambridge a KETpass = A2 = 2; PET Pass = B1 = 3, etc. With Cambridge exams tryalso, separately using the actual grades, treating an “A” at onelevel as the same as a “C” at the next level (i.e. as a “Pass” as thenext CEFR level); treat D/E as if it is a “pass” at the previous CEFRlevel.b. Create a “global” CEFR assessment result for each student byaveraging the CEFR Speaking/Writing Assessment results2. Correlate all these results to the test results and see which gives the highestcorrelation; use that at the external criterion. Plot correlation (Excel functionCORREL) of test to converted exam scores and to self-assessments; if thecorrelation is less than 0.7, collect more data on students at higher and atlower levels to extend range of level; if still less than 0.7: give up. If thecorrelation to self-assessments is higher than the correlation to the exams –then repeat procedure using self-assessments as the criterion. Then comparecut-offs from exam with “cut-offs from self-assessments and make acompromise.3. Create a “Scatterplot” (Excel: Insert Graph). Print copies of the plot and drawby hand a line of “best fit.”4. Create provisional cut-off scores between CEFR levels from the Scatterplot5. Use the cut-off scores to make a CEFR “Decision Table.”6. Calculate the percentage of matching classification7. Use the “Decision Table” to fine tune the “cut-off” scores in order to increasethis percentage of agreement.8. Create new “Decision Table9. Calculate percentage of agreement. Repeat cycle if necessary.© EAQUALS: The European Association for Quality Language Services:EAQUALS SECRETARIAT: P.O. Box 95, Budapest, H-1301 HungaryEmail: info@eaquals.org Internet: www.eaquals.orgRegistered Office: Via Torrebianca 18 34132 Trieste, Italy3

The European Association for Quality Language ServicesProcedureStep 1First convert examination results to CEFR levels, and convert/combine schools grades to give asingle CEFR result for each candidate. Then convert the CEFR levels to numbers, e.g. A1 = 1,A2 = 2, A2+ = 3, B1 = 4, B1+ = 5, etc.Step 2: Correlation.The next step is to enter the data in Microsoft Excel and calculate a correlation coefficientbetween the different external measures available and the results from the assessment understudy, in order to identify which is external measure best suited as external criterion. Yourschool Speaking/Writing assessment result is also an external measure.- Enter each student as a row in an Excel sheet.- Enter the assessment under study as the first column.- Enter the potential external criteria as further columns.- Place the cursor in an empty cell underneath the column with the first potential externalcriterion.- Select “Insert” “Function” and select or type in CORR. This activates the MS Excel wizard.- It asks you for “Array 1” – so click on the first cell of the data for the external criterion anddrag the cursor down to the last.- Repeat for “Array 2” with one of the potential external criteria.- Press “Okay” and hope for a high number.A weak correlation coefficient (e.g. 0.40) shows a weak relationship between the twoassessments. There is little point in proceeding further. There are all sorts of reasons why yourcorrelation might be low, but the better your test and the better the external criterion is atassessing the same thing, the higher the correlation will be. A strong correlation (e.g. 0.80)shows a very promising relationship between the two assessments. The technical significance ofthe size of the correlation coefficient depends upon the number of subjects.If you have no correlation reaching 0.70, then the relationship, whilst it may be significant, isnot really very strong. Solution: collect the same data on learners with higher and lower levels,add that in your table and try again. Adding more data around the top and the bottom of theability range of learners concerned extends the reporting scale and therefore almostautomatically increases the correlation. If doing this does not increase the correlation it is bestto give up, because either your test or your external criterion is just not very good. If you dohave some correlations above 0.7 and you can think of a plausible explanation for why thispotential external criterion is operating better than the others, then adopt that measure as yourexternal criterion and proceed to Step 3.Step 3: Creating a ScatterplotA correlation shows that there is a relationship. The next question is what precisely thatrelationship is. A table can be constructed that compares the CEFR level classification for eachperson on the assessment under study and on the assessment taken as a criterion measure.© EAQUALS: The European Association for Quality Language Services:EAQUALS SECRETARIAT: P.O. Box 95, Budapest, H-1301 HungaryEmail: info@eaquals.org Internet: www.eaquals.orgRegistered Office: Via Torrebianca 18 34132 Trieste, Italy4

The European Association for Quality Language ServicesFirst, however, we need to be able to set some provisional “cut-off scores” to at leasttentatively convert scores on the test under study to CEFR levels. If the test was developedfollowing academic testing conventions known as “standard-setting” then such cut-off scoreswill already exist. This current Step 2 is then a way of validating the cut-off scores developed inthat way. If however, the test has been developed by a school to test a certain level, but noscores have been allocated to different levels, we will first have to “guessestimate” some cutoffsas a working hypothesis in order to proceed further.Example Scatter plot5The Examination (Criterion)432100 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100My TestFigure 1: An example scatterplot with fictional dataTo do this, one need to compare the scores achieved on the test under study to the CEFR levelsreported by the external criterion, as first popularised by Brendan Carroll. 2 Figure 1 shows afictitious example. CEFR levels reported by an examination (external criterion) are shown up theleft hand axis (the Y axis), and scores on the test under study are shown along the bottom onthe “X axis.” Each point represents a student. This particular plot was created with the chart“Scatterplot” in MS Excel, but it could also be done on graph paper. The correlation, calculatedwith MS Excel as described in Step 1, is 0.74.Step 4: Setting Provisional Cut-off scoresNow a line of “best fit“ is drawn through the dots. Don’t lose time over this, it’s just a startingpoint. If it turns out that you drew the line in the wrong place, it can be corrected later in Step2 Carroll, B. J. (1980): Communicative Testing, Oxford, Pergamon Press: 61-64.© EAQUALS: The European Association for Quality Language Services:EAQUALS SECRETARIAT: P.O. Box 95, Budapest, H-1301 HungaryEmail: info@eaquals.org Internet: www.eaquals.orgRegistered Office: Via Torrebianca 18 34132 Trieste, Italy5

The European Association for Quality Language Services7. Figure 2 shows an example. This was drawn inn MS Word (having copied the chart fromExcel), but it could be done by hand on graph paper.Example Scatter plot5The Examination (Criterion)432100 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100My TestFigure 2: A line of “best fit”The next step is to draw a vertical line from the point at which the line of best fit touches thehorizontal lines (the reporting levels of the external criterion) down to the axis along the bottomwith the test scores. This is shown in Figure 3. These lines “cut” the X-axis. Anyone getting ascore between the point where the line of best fit cuts the horizontal line for “1” (representingA1) and the horizontal line “2” (representing A2) are A1 according to the test under study. Thisis the people who score above 20 (the “cut-score” or “cut-off score” for A1), and below 38, thecut-score A2. Here we have three people scoring 21, 23 and 32. Test and external criterionboth agree they are A1.Figure 3 would thus provide provisional cut-off scores approximately as follows:Level 1 (e.g. A1): 20-37Level 2 (e.g. A2): 38-56Level 3 (e.g. B1): 57-75Level 4 (e.g. B2): 76-(100)Level 5 (e.g. C1): We only had one person at this level so we really cannot say.© EAQUALS: The European Association for Quality Language Services:EAQUALS SECRETARIAT: P.O. Box 95, Budapest, H-1301 HungaryEmail: info@eaquals.org Internet: www.eaquals.orgRegistered Office: Via Torrebianca 18 34132 Trieste, Italy6

The European Association for Quality Language ServicesExample Scatter plot5The Examination (Criterion)432100 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100My TestFigure 3: Marking provisional “cut-scores”Step 5: Creating a Decision TableThe result gives us one person below Level 1 (A1) and 5 people at Level 1 (A1), two of whomhowever were classified as Level 2 (A2) by the examination used as an external criterion. AtLevel 2 (between 38 and 57) we have 5 people, 2 of whom were classified at Level 3 (B1) bythe examination. At the next Level things start to go a bit wonky (something not confined tofictional data). Our test has two people at Level 3, but the examination put them at Levels 1and 2 respectively.The two people the examination put at Level 3, our test has at Level 4. This relationshipbetween the two sets of scores can be expressed in what is called a “Decision Table 3 ,” asshown in Table 1There are 18 dots on Figures 1, 2 and 3 (our fictional data set) and therefore one sees a totalof 18 in the bottom right hand corner of Table 1.The number of people placed at each level on the examination (vertical axis) is shown abovethe number 18, those placed by the test (horizontal axis) along the bottom of the table. Theexamination placed 5 people at level 1 (A1): 3 of whom the test also placed there. These 3 arein the shaded square.3 The Decision Table in this form was recommended by Norman Verhelst (Cito) in the Council of EuropeManual for examination authorities. It is also known as a Bivariate Table or a Cross-classification Table.© EAQUALS: The European Association for Quality Language Services:7EAQUALS SECRETARIAT: P.O. Box 95, Budapest, H-1301 HungaryEmail: info@eaquals.org Internet: www.eaquals.orgRegistered Office: Via Torrebianca 18 34132 Trieste, Italy

The European Association for Quality Language Services0Test0 A1 A2 B1 B2 C1 TotalCriterion (Exams)A1A2B1B2C13 1 1 51 2 2 1 62 2 42 21 1Total 1 5 5 2 5 0 18Table 1: Decision Table (fictional data)Step 6: Calculating the Proportion of Common Classifications.Shading shows agreement between test and examination. The number of matchingclassifications is determined by adding together those learners shown in the shaded diagonal,as in Table 2.A1 A2 B1 B2 C1 Total3 + 2 + 0 + 2 + 0 = 7Table 2: Table 16 Common classifications to CEFR levels7 out of 18 = 39% common classifications.Step 7: Fine-tuning cut-scoresLooking at Table 1 again, there is one point that does suggest an imbalance: 3 of the peoplewhom the examination placed at Level 2 (A2) have come out at only Level 1 (A1) or below onthe test. Possibly the cut-point for Level 2 (A2) is set too high. Certainly there are two learnerswho scored 33 and 35 respectively who just missed the cut-score of 38. Perhaps we shouldlower the cut-scores for A1 and A2 and then collect some more data on low level learners tocheck.Adjusting cut-scores in this way would give us cut-scores as follows:Level 1 (e.g. A1): 15-32Level 2 (e.g. A2): 33-56Level 3 (e.g. B1): 57-75Level 4 (e.g. B2): 76-(100)© EAQUALS: The European Association for Quality Language Services:EAQUALS SECRETARIAT: P.O. Box 95, Budapest, H-1301 HungaryEmail: info@eaquals.org Internet: www.eaquals.orgRegistered Office: Via Torrebianca 18 34132 Trieste, Italy8

The European Association for Quality Language ServicesStep 8: Adjust Decision TableThe revised cut-off scores can be plotted in a new Decision Table as in Table 3. The pictureshown by Table 3 does now look more balanced. Obviously the fictional learner who failed toreach A1 on the test had an exceptionally bad day. If we were to do a sophisticated statisticalanalysis of the test we would no doubt discover that their responses were inconsistent.Otherwise our problem is the lack of “hits” at B1. But it is balanced.0Test0 A1 A2 B1 B2 C1 TotalCriterion (Exams)A1A2B1B2C13 1 1 51 4 1 62 2 42 21 1Total 1 3 7 2 5 0 18Table 3: Decision Table (fictional data) - revisedStep 9: Recalculating the Proportion of Common Classifications.As Table 3 shows, we now have 9 matched classifications – or 50%. This is a very respectableresult.A1 A2 B1 B2 C1 Total3 + 4 + 0 + 2 + 0 = 9Table 4: Table 3 Common classifications to CEFR levelsFinally, let us repeat the procedure with a real example.Table 5 shows how teacher judgements were used to confirm provisional cut-off pointsbetween levels on the Eurocentres Scale of Language Proficiency for an item bank testingsystemic knowledge of German. Both sets of results are presented in terms of levels on theEurocentres scale, not the CEFR. The study is illustrated here only as a publishedmethodological example undertaken in a language school 4 . The results on the two assessmentsare simply plotted against each other in a Microsoft Word table. Again it helps to shade thediagonal line – which is where all the learners would be if the correlation between the twoassessments was absolutely perfect.4 North, B. (2000): Linking Language Assessments: an example in a low stakes context. System 28, 555-577.© EAQUALS: The European Association for Quality Language Services:EAQUALS SECRETARIAT: P.O. Box 95, Budapest, H-1301 HungaryEmail: info@eaquals.org Internet: www.eaquals.orgRegistered Office: Via Torrebianca 18 34132 Trieste, Italy9

The European Association for Quality Language ServicesDespite an extremely high correlation of 0.93, only 28 of the 68 subjects (41%) have actuallyreceived exactly the same level on the 9 Eurocentres levels. The reason why so few learnershave received the same classification concerns the number of levels on the scale. TheEurocentres scale splits each CEFR level other than A1 and C2 into two. Thus Levels 2 & 3 makeA2; Levels 4 & 5 make B1; Levels 6 & 7 make B2, and Levels 8 & 9 make C1. It is easier to geta higher number of correct classifications with a smaller number of levels. Using finer levels (asEurocentres do) may be pedagogically meaningful, but in a testing context it requires a higherdegree of decision power.I 9 2t 8 1e 7 8 3m 6 2 8b 5 1 8 2 1a 4 4 4n 3 2 6 5k 2 1 1 5e 1 4r 1 2 3 4 5 6 7 8 9Teacher Judgement: SprachkenntnisseTable 5: Referencing an Item Bank to Teacher JudgementsThis problem of the weaker decision power and greater number of mis-classifications with alarger number of reporting levels is the reason “high stakes” assessment (examinations) tendsto use broad levels. This is also why the CEFR and the European Language Portfolio have only 6criterion levels.Whatever set of finer level classifications may be used locally, only the official 6 CEFR levels A1-C2 should be used when aligning assessments to the CEFR. Table 6 therefore shows a “DecisionTable” using only CEFR levels as recommended.© EAQUALS: The European Association for Quality Language Services:EAQUALS SECRETARIAT: P.O. Box 95, Budapest, H-1301 HungaryEmail: info@eaquals.org Internet: www.eaquals.orgRegistered Office: Via Torrebianca 18 34132 Trieste, Italy10

The European Association for Quality Language Servicesdemonstrated above. And the extent to which two different measures correlate will always belimited. Correlations of 0.7-0.8 between two good, validated tests or examinations are normal.The equivalences published by Educational Testing Services on the equivalent scores for TOEFLand TOEIC were long based on a study showing a correlation of only 0.75 between the twotests. TOEFL and TOEIC have different styles and test language in different domains. Otherpairs of tests will use different item types. Performance tests (teacher rating of a sample)assess something different to knowledge tests (teacher marking of language items). Teachers –over a period of time – will see things that tests – taken one morning - do not.Apart from such “legitimate” reasons why results from different tests will only relate in a limitedfashion, there are all sorts of other reasons. Testing is an imperfect science. Learners are notinvariable objects: atmosphere, emotional mood, amount of sleep, digestion – all contextualfactors affect assessment results.Therefore, if one can achieve a modest success in independently proving a statisticalrelationship between an assessment and the CEFR levels by following logical steps in aprincipled manner as suggested, then one should be more than satisfied.© EAQUALS: The European Association for Quality Language Services:EAQUALS SECRETARIAT: P.O. Box 95, Budapest, H-1301 HungaryEmail: info@eaquals.org Internet: www.eaquals.orgRegistered Office: Via Torrebianca 18 34132 Trieste, Italy12