Guide for Developing Assessment Tools - National Skills Standards ...

NQC | Guide for developing assessment tools3. Quality ChecksThere are several checks that could be undertaken (as part of the quality assuranceprocedures of the organisation) prior to implementing a new assessment tool. For example,the tool could be:• Panelled with subject matter experts (e.g. industry representatives and/or other colleagueswith subject matter expertise) to examine the tool to ensure that the content of the tool iscorrect and relevant. The panellist should critique the tool for its:Clarity;Content accuracy;Relevance;Content validity (i.e. match to unit of competency and/or learning outcomes);Avoidance of bias; and/orAppropriateness of language for the target population.• Panelled with colleagues who are not subject matter experts but have expertise inassessment tool development. Such individuals could review the tool to check that it has:Clear instructions for completion by candidates;Clear instructions for administration by assessors; andAvoidance of bias.• Piloted with a small number of individuals who have similar characteristics to the targetpopulation. Those piloting the tool should be encouraged to think out aloud whenresponding to the tool. The amount of time required to complete the tool should berecorded and feedback from the participants should be gathered about the clarity of theadministration instructions, the appropriateness of its demands (i.e. whether it is toodifficult or easy to perform), its perceived relevance to the workplace etc.• Trialled with a group of individuals who also have similar characteristics to the targetpopulation. The trial should be treated as though it is a dress rehearsal for the ‘realassessment’. It is important during the trial period that an appropriate sample size isemployed and that the sample is representative of the expected levels of ability of thetarget population. The findings from the trial will help predict whether the tool would:Be cost effective to implement; Be engaging to potential candidates;Produce valid and reliable evidence;Be too difficult and/or too easy for the target population; Possibly disadvantage some individuals;Able to produce sufficient and adequate evidence to address the purpose of the assessment; as well asGUIDE FOR DEVELOPING ASSESSMENT TOOLS PAGE 18