A Readiness for Clerkship Survey: Can Self- Assessment Data be ...

A Readiness for Clerkship Survey: Can Self-
Assessment Data be Used to Evaluate
Program Effectiveness

Authors
EVALUATION STUDIES UNIT

What’s new
EVALUATION STUDIES UNIT

Self-Assessment-Accuracy
Invalid Predictor of Individual Performance
• Woolliscroft et al. Acad Med 68(4):285-294,1993
• Ward, Gruppen and Regehr. Adv Health Sci Educ
Theory Pract. 7:63-80, 2002
• Weiss et al. Med Teach 27(5):445-449,2005
• Davis et al. JAMA 296:1094-1102, 2006
EVALUATION STUDIES UNIT

Self-Assessment Data
“Wisdom of Crowds” Surowiecki 2004
Can accurate information be gained by
aggregating many flawed opinions
EVALUATION STUDIES UNIT

Self-Assessment Data
“Wisdom of Crowds” Surowiecki 2004
Can accurate information be gained by
aggregating many flawed medical student
opinions
EVALUATION STUDIES UNIT

Gaps in Program Evaluation…
• Generally limited to:
• student opinion of the quality of individual courses,
clerkships, longitudinal courses and curricular
components
• student performance data and other student-related
outcome data
• “The whole is greater than the sum of its parts”
• No instruments to evaluate the function of the preclerkship
curriculum as a whole in preparing students
for clerkship
• Faculty input is seldom collected
EVALUATION STUDIES UNIT

The Call for Competency-Based
Education
• Harden 2002, Developments in outcome-based
education. Med Teacher 24(2): 117-120.
• CanMeds 2005
• AFMC Future of Medical Education in Canada 2010
• “Educating Physicians: a call for reform of medical
school and residency 2010”
EVALUATION STUDIES UNIT

Readiness for Clerkship Survey
Competency-based
Represents the competencies that the program
should enable students to achieve
Instrument was blueprinted/developed using
several documents which adequately capture
educational goals of the preclerkship program
• UBC MD Program Graduate Competencies
EVALUATION STUDIES UNIT

Competency-based
Readiness for Clerkship Survey
List of action statements that articulate
competencies in:
• cognitive skills
• clinical skills
• professional behaviours
Mapped back to the source documents
Addressed gaps and redundancies
Expert review
EVALUATION STUDIES UNIT

Readiness for Clerkship Survey
• Student version – self-assessment
• Please rate your ability, i.e., competence to do
or perform the activities listed below...
• Faculty version – observe/assessor
• Please rate the level of competency, as a group,
of NEW third year UBC medical students at the
beginning of the rotation to do or perform the
activities listed below...
• 39 items in order of a typical patient
encounter
EVALUATION STUDIES UNIT

Survey AFTER starting clerkship
Sent 3-5 months after the start of Year 3
• Third year students
• 2011 (n = 179; 68%)
• 2012 (n = 171; 64%)
• Faculty
• 2011 (n = 384; 26%)
• 2012 (n = 419; 39%)
EVALUATION STUDIES UNIT

How should we analyze the data
Discriminate between raters
Discriminate between the items
EVALUATION STUDIES UNIT

Generalizability studies
Students
Faculty
2011 2012 2011 2012
Item 16.10 31.82 7.90 13.31
Rater 28.78 27.22 48.56 39.90
Item x Rater 55.12 40.96 43.54 46.80
EVALUATION STUDIES UNIT

Decision studies
Are students able to reliably differentiate between
aspects of competence they, as a group, possess
Students *
G G(n) k
2011 (n = 137) 0.16 0.96 21
2012 (n = 104) 0.32 0.98 9
Faculty †
2011 (n = 89) 0.08 0.88 45
2012 (n = 104) 0.13 0.94 26
EVALUATION STUDIES UNIT

How accurate are students’
judgments in aggregate
Student vs Faculty 2011 2012
r 0.88 0.91
• Students rated themselves higher
• 38/39 items 2011 and 34/39 items 2012 p

Two Subscales Identified
1. Clinical skills and Knowledge Application
• Obtaining and interpreting information from and about the
patient, and integrating this with prior knowledge leading to the
formulation of a diagnosis and management.
2. Working as a professional
• Being responsible for and interacting with the patient and team,
and demonstrating self-care and self-directed learning.
Subscale reliability ranged from .90 to .94
EVALUATION STUDIES UNIT

Summary
• Aggregated student self-assessments:
• Reliably differentiate between different aspects of
competence
• Correlate well with faculty data
• Require fewer students than faculty
• Identify strengths and weaknesses of the
program
• Survey items cluster into meaningful constructs
that align with a competency framework
EVALUATION STUDIES UNIT

Limitations
Faculty response rates
Do not reflect the absolute competence of the
group
Are strengths and weaknesses unique to our
program
EVALUATION STUDIES UNIT

Parting Comment
Readiness survey can be used to evaluate
the function of the pre-clerkship curriculum
• Survey should be tailored to reflect the
expected competencies
EVALUATION STUDIES UNIT

Thank you
Linda.peterson@ubc.ca
EVALUATION STUDIES UNIT

Rank Order 10 Lowest
EVALUATION STUDIES UNIT

Rank Order 10 Highest
EVALUATION STUDIES UNIT

Internal Consistency of the instrument
Do items cluster into meaningful constructs
that align with a competency framework
Student/faculty and cohort comparisons
Factor analysis: exploratory and confirmatory
EVALUATION STUDIES UNIT