GME Evaluation Task Force Recommendation - UCSF School of ...

UCSF Office of Graduate Medical Education
GME Evaluation Task Force
Report and Recommendations
July 1, 2008
UCSF School of Medicine
Graduate Medical Education
500 Parnassus Ave. MU 250E
San Francisco, CA 94143-0474
www.medschool.ucsf.edu/gme

June 2008
Preface
UCSF Offi ce of Graduate Medical Education
The ACGME Outcome Project was initiated in 2000 and is currently in Phase 3 of its implementation.
The goal for Phase 3 is “full integration of the competencies and their assessment with learning
and patient care.” Programs are required to use a minimum of 2 methods for evaluating each of
the 6 ACGME general competencies and to use resident performance data as the basis for program
improvement.
Implementing robust assessment systems for the 6 competencies has proved diffi cult for most
residency programs. Logistical challenges and a lack of centralized tools are primarily responsible.
The ACGME has provided guidance in the form of a Toolbox, Think Tank Recommendations and, most
recently, Implementation Guides. These materials are available at: htt p://www.acgme.org/outcome/.
However, these resources off er annotated lists of recommended assessment tools and do not propose
a core set to be used across GME programs.
In mid-2007 the Associate Dean for Graduate Medical Education appointed a GME Evaluation Task
Force to be chaired by the GME Director of Curricular Aff airs. The Task Force was charged to “use its
theoretical and practical expertise to review currently used tools and propose a centralized set that all
GME programs will be encouraged to use. The set will include assessments of resident competency
that are common across programs, evaluations of faculty teaching, and approaches for benchmarking
program performance.”
In making our recommendations the Task Force aims to supplement the ACGME resources by
identifying best practices at UCSF and proposing a core set of tools for the assessment of competencies
shared across all GME programs at UCSF: professionalism, interpersonal & communication skills,
and elements of practice based learning and improvement.
The Task Force met 1-2 times each month from September 2007 to June 2008 and searched for best
practices among the tools used by residency and fellowship programs at UCSF. We applied our
knowledge of the literature and evidence on the reliability and validity of assessment in medical
education to help guide the recommendations. Theoretical considerations were especially helpful
when evidence was lacking or incompletely developed. Dr. Lee Learman, Task Force Chair, wishes to
thank its members for their many hours of hard work and excellent collaboration: Patricia O’Sullivan,
Ed.D. and Arianne Teherani, Ph.D. (Offi ce of Medical Education), Sumant Ranji, M.D. (Department of
Medicine), and Gitanjali Kapur (GME). We also thank two consultants to the Task Force, Drs. Susan
Promes (Emergency Medicine) and John Young (Psychiatry), and Laura Pliska, Ob/Gyn Residency
Program Manager, who generously contributed workfl ow documents to the Task Force.
2

June 2008
Table of Contents
UCSF Offi ce of Graduate Medical Education
General Considerations 4
Annotated Glossary of Evaluation Tools 5
Core Measures for UCSF GME 7
• Global Assessments
8 - 12
•
•
•
Patient Care
Mini-Clinical Evaluation Exercise (Mini-CEX) 13 - 16
Focused (Checklist) Assessments 17 - 19
Professionalism, Interpersonal & Communication Skill
Health Care Team and Self Evaluations 20 - 26
Patient Surveys 27 - 31
Practice-based Learning and Improvement
Critical Appraisal 32 - 34
Clinical Teaching 35 - 44
Additional Recommendations 45
• Medical Knowledge
46 - 47
• Systems-based Practice
48 - 50
• Progress Report for Semi-Annual Review
51 - 52
• Closing the Loop: Annual Program Review
53
Confi dential Resident Evaluation of Faculty Teaching 54 - 64
Program Evaluation by Faculty and Residents 65 - 66
Progress Report for Annual Program Review 67 - 68
APPENDICES
A. ACGME Common Program Requirements IVB: General Competencies 69 - 70
B. ACGME Common Program Requirements V: Evaluation 71 - 72
C. Examples of Focused Assessment Tools 73 - 81
3

June 2008
General Considerations
UCSF Offi ce of Graduate Medical Education
The ACGME has defi ned 6 general competencies for residents that must be integrated into the curriculum
and evaluated using objective assessments and multiple evaluators. Although many programs and
institutions regard their trainee’s clinical teaching skills as an important 7th competency, participating
in the education of patients, families, students, residents and other health professionals is actually a
component of practice-based learning and improvement. The ACGME General Requirements for
curriculum and evaluation can be found in Appendices A and B. Like the ACGME, when we use the
term ‘resident’ in our report we refer to both UCSF residents and fellows.
The ACGME requires each competency to be assessed using a minimum of 2 diff erent tools. To assist
programs in compliance with the requirements and improve uniformity in our assessments at UCSF, the
Task Force proposes a set of core measures to be used across GME programs at UCSF. Training programs
are encouraged to add additional items to these tools to address the specifi c objectives of their programs
and to use additional tools when necessary.
The assessments we conduct are formative in that they are used primarily by residents and their mentors
to provide feedback and develop future learning plans and goals. According to the ACGME summative
assessment occurs at completion of the training program, at which time the program director must verify
that the resident has demonstrated suffi cient competence to enter practice without direct supervision
It is important to distinguish our Task Force’s work from the implementation of electronic portfolios for
learner assessment at UCSF School of Medicine. A portfolio is a purposeful and longitudinal collection of
tangible evidence of learner-selected work that exhibits the learner’s eff orts, progress or achievement. The
portfolio features the criteria for selection and judging merit, and includes evidence of learner refl ection.
In this context, evaluation tools provide ‘tangible evidence’ of learning. However, even the most robust
assessment system should not be regarded as a portfolio without the key elements of learner-centered
work and evidence of learner refl ection.
This report includes a series of short guides with evaluation tools and workfl ow documents to help
programs implement the assessments. We developed each guide to be a stand-alone module including
only the most essential information for program directors and coordinators. The Task Force discourages
programs from implementing an evaluation tool without fi rst considering the recommendations included
in the corresponding guide.
Our report also includes recommendations for the evaluation of medical knowledge and systems based
practice, a resident progress report for semi-annual, review, and recommendations for conducting the
annual program review including assessment of faculty teaching and a program progress report.
The guides and tools are accessible via the GME website and the GME E*Value system.
Questions about the tools and guides should be directed to:
Gitanjali Kapur, GME Educational Technologies Analyst: kapurg@medsch.ucsf.edu
GME Evaluation Handbook: htt p://medschool.ucsf.edu/gme/curriculum/evaltools.html
4

June 2008
Annotated Glossary of Evaluation Tools
UCSF Offi ce of Graduate Medical Education
The ACGME/ABMS Toolbox, ACGME Think Tank Recommendations, and Implementation Booklets
are available at htt p://www.acgme.org/outcome and describe the range of potential assessment
methods. The summaries below are excerpted and adapted from the ACGME materials and include
only the types of tools being recommended by the Evaluation Task Force.
GLOBAL RATING FORMS: Usually completed by faculty supervisors at the end of a clinical rotation
assignment, global forms ask judges to rate trainees on general categories of ability (e.g., patient
care skills, communication skills, medical knowledge) rather than specifi c tasks, skills or behaviors.
Global ratings are completed retrospectively and are based on general impressions collected over a
period of time. As such, they are subject to subjectivity, recall bias, and halo eff ects in which positive
or negative impressions of the trainee infl uence the specifi c ratings. Because they are relatively
easy to collect, global rating forms are nearly ubiquitous in GME. Unfortunately, they do not yield
suffi ciently reliable or valid data for assessment of the competencies and must be supplemented
with a second, bett er measure. Another kind of global rating form is used by learners to assess their
clinical educators at the end of a learning experience (rotation, continuity clinic, etc.).
WRITTEN EXAMINATIONS: These are usually composed of multiple-choice questions (MCQ)
selected to sample medical knowledge and understanding of a defi ned body of knowledge, not just
factual or easily recalled information. Each question or test item contains an introductory statement
followed by four or fi ve options in outline format. The examinee selects one of the options as the
presumed correct answer by marking the option on a coded answer sheet. Only one option is keyed
as the correct response. The introductory statement oft en presents a patient case, clinical fi ndings, or
displays data graphically. The in-training examinations prepared by specialty societies and boards
use MCQ type test items. A typical half-day examination has 175 to 250 test questions. Comparing
the test scores on in-training examinations with national statistics can serve to identify strengths
and limitations of individual residents to help them improve. Comparing test results aggregated
for residents in each year of a program can be helpful to identify residency training experiences that
might be improved.
360 DEGREE EVALUATION: 360-degree evaluations consist of measurement tools completed by
multiple people in a person’s sphere of infl uence. Evaluators completing rating forms in a 360-degree
evaluation usually are superiors, peers, subordinates, and patients and families. Most 360-degree
evaluation processes use a survey or questionnaire to gather information about an individual’s
performance on several topics. Data are then shared with the learner, who compares the perspectives
of others with their self-assessment of the same qualities. 360 assessments are most useful for the
5

June 2008
assessment of professionalism and interpersonal & communication Skills.
UCSF Offi ce of Graduate Medical Education
FOCUSED OBSERVATION: Checklist Evaluations evaluate the essential or desired specifi c
behaviors, activities, or steps that make up a more complex competency or competency component.
Checklists can be tailored to assessed detailed actions for a specifi c task. Typical response options
on these forms are a check (¬) or “yes” to indicate that the behavior occurred or options to indicate
the completeness (complete, partial, or absent) or correctness (total, partial, or incorrect) of the
action. Checklists are useful for evaluating any competency and competency component that can
be broken down into specifi c behaviors or actions. Checklists can be used as reliable measures of
patient care (e.g., surgical or other procedural skills) , interpersonal and communication skills
(e.g., informed consent, disclosure of adverse outcome), and practice-based learning (e.g., critical
appraisal of evidence).
Another type of assessment based on direct observation is the Mini-Clinical Evaluation Exercise
(Mini-CEX). The Mini-CEX is completed immediately aft er observing a patient encounter. However,
instead of using skill checklists the raters complete scales covering important domains of patient
care including interviewing and physical examination skills, humanistic qualities, counseling skills
and clinical judgment. If enough observations are performed the Mini-CEX can be used as a reliable
6

EVALUATION OF THE ACGME COMPETENCIES:
CORE MEASURES FOR GME AT UCSF
7
UCSF GME ‐ 7/08

GME Evaluation Task Force Recommendation
GLOBAL ASSESSMENTS
Despite their common usage and familiarity, global assessments are considered weak
evidence of learning. If used along with an evaluation tool of greater validity, global
assessments may be used to evaluate all 6 ACGME competencies.
Recommended Assessment Tool
Many UCSF programs use a short global assessment tool that includes the minimum
language version of the 6 competencies and a 9 point scale. This format is preferred to a
longer assessment form listing the full language version of each competency in multiple
items. The 9‐point scale is preferred to shorter scales because it allows raters to identify
differences among residents who meet or exceed expectations. An additional item
assessing overall performance is also recommended. A sample global assessment is
provided below.
Reliability and Validity
Global assessments are subject to recall bias and halo effects which threaten validity and
imprecision which threatens reliability. Given these limitations the least burdensome
possible tool should be used. Greater numbers of evaluators may improve inter‐rater
reliability but will not make up for the subjective nature of these assessments.
Administration
o Timing: after each clinical rotation; for continuity clinics at least twice annually.
o Who Performs: only individuals who directly supervised the resident during the
specific timeframe being assessed.
o Format: Each competency is represented by one item rated on a 9‐point scale
anchored with descriptive language on either end and divided into thirds (1‐3:
unsatisfactory / does not meet expectations, 4‐6: satisfactory / meets expectations,
7‐9: outstanding / exceeds expectations). An additional item assesses overall
competency. Written comments are provided listing strengths and opportunities
for improvement. Programs should review progress and share feedback with
residents midway and after each rotation. Mid‐rotation feedback is especially
important for residents who appear not to be meeting criteria for satisfactory
performance.
o Scoring Criteria and Training: The distinction between satisfactory and
unsatisfactory performance is an important one. If guidelines are not available
for making this distinction, standard‐setting can be used to improve accuracy.
Standard‐setting is especially important for global assessments because raters
tend to fall into stricter vs. lenient categories.
8
UCSF GME ‐ 7/08

o Documentation: Score summaries are generated automatically by the evaluation
management system and can be accessed by residents after each rotation. Overall
progress should be reviewed in writing at least twice annually.
Uses of the Data
o Comparing global evaluation scores to performance of peers or one’s own prior
performance can help identify trends.
o Summative Decisions: Global evaluation scores persistently falling below
expectations (3 or less on the 9 point scale) could delay or prevent a resident from
advancing or graduating. Generally, however, such decisions would be based on
overall assessments of progress incorporating other mearues.
o Remediation Threshold: Programs should communicate what performance on
global assessments would trigger remediation. For example, residents and
faculty would be informed in advance that scores < 4 are unsatisfactory and a
score of 4 is marginal.
9
UCSF GME ‐ 7/08

University of California, San Francisco
Core Measure for UCSF GME: Global Assessment Form
Subject:
Evaluator:
Site:
Period:
Dates of Activity:
Activity:
Evaluation Type: Resident
Faculty Contributing to Evaluations (Question 1 of 12 - Mandatory)
Patient Care (Question 2 of 12 - Mandatory)
In evaluating this resident’s performance, use as your standard the level of knowledge, skills and attitudes expected from a clearly
satisfactory resident at this stage of training
Insufficient
contact to
judge
Incomplete, inaccurate medical interviews,
physical examinations, and review of other
data; incompetent performance of essential
procedures; fails to analyze clinical data and
consider patient preferences when making
decisions.
Needs
Improvement
(1:3 require
comment)
Needs
Improvement
Needs
Improvement
Meets
Expectations
Meets
Expectations
10
Superb, accurate, comprehensive medical
interviews, physical examinations, review of
other data, and procedural skills; always
makes diagnostic and therapeutic decisions
based on available evidence, sound
judgment, and patient preferences.
Meets
Expectations
Exceeds
Expectations
Exceeds
Expectations
Exceeds
Expectations
0 1 2 3 4 5 6 7 8 9
Medical Knowledge (Question 3 of 12 - Mandatory)
In evaluating this resident’s performance, use as your standard the level of knowledge, skills and attitudes expected from a clearly
satisfactory resident at this stage of training
Insufficient
contact to
judge
Limited knowledge of basic and clinical
sciences; minimal interest in learning; does
not understand complex relations,
mechanisms of disease.
Needs
Improvement
(1:3 require
comment)
Needs
Improvement
Needs
Improvement
Meets
Expectations
Meets
Expectations
Exceptional knowledge of basic and clinical
sciences, highly resourceful development of
knowledge; comprehensive understanding
of complex relationships, mechanisms of
disease.
Meets
Expectations
Exceeds
Expectations
Exceeds
Expectations
Exceeds
Expectations
0 1 2 3 4 5 6 7 8 9
Practice-Based Learning Improvement (Question 4 of 12 - Mandatory)
In evaluating this resident’s performance, use as your standard the level of knowledge, skills and attitudes expected from a clearly
satisfactory resident at this stage of training
Insufficient
contact to
judge
Fails to perform self-evaluation; lacks
insight, initiative; resists or ignores
feedback; fails to use information technology
to enhance patient care or pursue selfimprovement.
Needs
Improvement
(1:3 require
comment)
Needs
Improvement
Needs
Improvement
Meets
Expectations
Meets
Expectations
Constantly evaluates own performance,
incorporates feedback into improvement
activities; effectively uses technology to
manage information for patient care and
self-improvement.
Meets
Expectations
Exceeds
Expectations
Exceeds
Expectations
Exceeds
Expectations
0 1 2 3 4 5 6 7 8 9
UCSF GME ‐ 7/08

Interpersonal & Communication Skills (Question 5 of 12 - Mandatory)
In evaluating this resident’s performance, use as your standard the level of knowledge, skills and attitudes expected from a clearly
satisfactory resident at this stage of training
Insufficient
contact to
judge
Does not establish even minimally effective
therapeutic relationships with patients and
families; does not demonstrate ability to
build relationships through listening,
narrative or nonverbal skills; does not
provide education or counseling to patients,
families or colleagues.
Needs
Improvement
(1:3 require
comment)
Needs
Improvement
Needs
Improvement
Meets
Expectations
Meets
Expectations
11
Establishes a highly effective therapeutic
relationship with patients and families;
demonstrates excellent relationship building
through listening, narrative and nonverbal
skills, excellent education and counseling of
patients, families, and colleagues.
Meets
Expectations
Exceeds
Expectations
Exceeds
Expectations
Exceeds
Expectations
0 1 2 3 4 5 6 7 8 9
Professionalism (Question 6 of 12 - Mandatory)
In evaluating this resident’s performance, use as your standard the level of knowledge, skills and attitudes expected from a clearly
satisfactory resident at this stage of training
Insufficient
contact to
judge
Lacks respect, compassion, integrity,
honesty; disregards need for selfassessment;
fails to acknowledge errors,
does not consider needs of patients,
families, colleagues; does not display
responsible behavior.
Needs
Improvement
(1:3 require
comment)
Needs
Improvement
Needs
Improvement
Meets
Expectations
Meets
Expectations
Always demonstrates respect, compassion,
integrity, honesty, teaches/role models
responsible behavior; total commitment to
self-assessment; willingly acknowledges
errors; always considers needs of patients,
families, colleagues.
Meets
Expectations
Exceeds
Expectations
Exceeds
Expectations
Exceeds
Expectations
0 1 2 3 4 5 6 7 8 9
Systems-Based Learning (Question 7 of 12 - Mandatory)
In evaluating this resident’s performance, use as your standard the level of knowledge, skills and attitudes expected from a clearly
satisfactory resident at this stage of training
Insufficient
contact to
judge
Unable to access/mobilize outside
resources; actively resists efforts to improve
systems of care; does not use systematic
approaches to reduce error and improve
patient care.
Needs
Improvement
(1:3 require
comment)
Needs
Improvement
Needs
Improvement
Meets
Expectations
Meets
Expectations
Effectively accesses/utilizes resources;
effectively uses systematic approaches to
reduce errors and improve patient care;
enthusiastically assists in developing
systems improvement.
Meets
Expectations
Exceeds
Expectations
Exceeds
Expectations
Exceeds
Expectations
0 1 2 3 4 5 6 7 8 9
Educational Objectives (Question 8 of 12 - Mandatory)
Resident’s Achievement of Educational Objectives on This Rotation
Insufficient
contact to
judge
Needs
Improvement
(1:3 require
comment)
Needs
Improvement
Needs
Improvement
Meets
Expectations
Meets
Expectations
Meets
Expectations
Exceeds
Expectations
Exceeds
Expectations
Exceeds
Expectations
0 1 2 3 4 5 6 7 8 9
Strengths (Question 9 of 12 - Mandatory)
UCSF GME ‐ 7/08

Opportunities for Improvement (Question 10 of 12)
System Ease of Use (Question 11 of 12)
E*Value was easy to use.
NA Strongly Disagree Disagree Neutral/Undecided Agree Strongly Agree
0 1 2 3 4 5
E*Value Comments: (Question 12 of 12)
Comments entered here will be forwarded to E*Value technical support and will not be anonymous.
12
UCSF GME ‐ 7/08

GME Evaluation Task Force Recommendation
PATIENT CARE
PATIENT CARE SKILLS (MINI‐CEX)
Recommended Assessment Tool
The Mini‐Clinical Evaluation Exercise (Mini‐CEX) was developed and studied
extensively by the American Board of Internal Medicine. It is a focused assessment on
specific aspects of a patient interaction. As such it can assess principles of patient care
foremost and secondarily it asks for ratings of professionalism and interpersonal and
communication skills as these are important components of every patient interaction.
Reliability and Validity
For use at the semi‐annual review meeting a minimum of 6 forms/year would provide
satisfactory reliability with 12 being optimal. The reliability and validity is based on the
research done with the Mini‐CEX. The program must monitor that the forms are being
completed correctly including signatures to ensure that they are measuring with the
same psychometric rigor as was done in the research studies.
Administration
o Timing: 6 (up to 12) assessments per year
o Who Performs: Skill assessments should be done through observation of the
actual performance. It is possible that a faculty member could assess through
video review of the performance, but the assessment reflects the skill of the
resident on the performance date. The entire clinical encounter does not need to
be observed – a shorter duration of observation may be more efficient.
o Format: A checklist is the most appropriate format for evaluating specific
procedural or communication skills (see section on focused assessment of
observed skills). Because the Mini‐CEX is developed for generic use across
different encounter types, it uses scales to assess important elements in any
encounter (e.g., history‐taking, the physical examination, humanistic qualities and
clinical reasoning).
o Scoring Criteria and Training: The Mini‐CEX research has been done on the 9
point scale for assessment. While other research may suggest that fewer points
will give the same decisions, it is recommended with maintaining the 9 point
scale as designed. The form includes a glossary describing the skills being
assessed but no criteria are provided for unsatisfactory vs. satisfactory vs.
superior performance. Attending faculty review the form and it is considered self‐
explanatory. This is not ideal, but most expedient.
o Documentation: At minimum, twice annually as part of semi‐annual review
meetings.
13
UCSF GME ‐ 7/08

Uses of the Data
o Formative Feedback: Concurrent, written same‐day feedback is recommended.
The Mini‐CEX is an observational form and must be completed in real time. It sets
the expectation that resident and faculty member will discuss the observation and
sign the form. More details may be found in an article by Holmboe (1).
o Summative Decisions: Programs should inform residents that the criteria for
judging progress will be the performance on the Mini‐CEX averaged across all
observations. The program should indicate a standard that would generate action
such as any single Mini‐CEX with an ʺunsatisfactoryʺ rating. These decisions
criteria should be made explicit to the residents.
o Remediation Threshold: Programs should communicate what performance on the
Mini‐CEX would require remediation. However, the faculty must be willing to
support such a process or they may be likely to inflate performance so as to not be
burdened with remediation. Most programs would consider an average of 5 or
below on the Mini‐CEX while supposedly indicating satisfactory performance
worthy of a development plan for the resident.
o Program Effectiveness: The Mini‐CEX is so intertwined with the fundamentals of
patient care that the data are to assess resident performance and generate plans as
needed. They are less likely to be useful for program effectiveness.
References
1. Holmboe ES, Yepes M, Williams F, Huot SJ. Feedback and the Mini Clinical
Evaluation Exercise. J Gen Intern Med 2004; 19(5 Pt 2): 558–561.
14
UCSF GME ‐ 7/08

15
UCSF GME ‐ 7/08

16
UCSF GME ‐ 7/08

GME Evaluation Task Force Recommendation
PATIENT CARE
PATIENT CARE: FOCUSED ASSESSMENT OF OBSERVED SKILLS
Evidence of Competency In:
Surgical Skills, Procedural Skills, Specific Communication Skills
Recommended Assessment Tool
Unlike the shared general competencies, Patient Care objectives are defined by each
RRC. Thus, no specific tool can be recommended across GME programs. Instead, we
offer principles to guide the development or selection of tools. Checklists define the
discrete tasks that comprise the overall skill being assessed. Scales evaluate procedural
skills that transcend individual tasks, such as knowledge of anatomy, instrument
handling, flow of operation, etc. Richard Reznick and colleagues at the University of
Toronto developed scales for the Observed Structured Assessment of Technical Skills, or
O‐SATS (1,2).
Examples of skill checklists are included in the Appendix.
Overview
Focused assessments for skills are used primarily to assess patient care by determining if
a psychomotor skill has been acquired. It is possible to combine some communication
items if this is the only time a resident interacts with patients, but it is not the primary
use of this assessment. It is also possible to develop a focused assessment of specific
communication skill tasks, such as an informed consent discussion or specific counseling
following a practice guideline.
Reliability and Validity
Skill checklists primarily have content validity. The items for a specific checklist may
have come from the literature where someone has decided the checklist has content
validity. Alternatively, if designing a checklist within the program, having those who
are “expert” in the skill review and approve the checklist would serve as a level of
validity. Reliability exists in several dimensions: the ability of different assessors to
come to the same decision (inter‐rater reliability) and the internal consistency of the
checklist items (do they “fit” together). However, if checklists are used to identify when
someone has mastered the skill, the internal consistency is not relevant (since everyone
should get 100% eventually). Therefore, the best reliability evidence would be
consensus of faculty concerning the decision that the resident is competent in that skill.
17
UCSF GME ‐ 7/08

Administration
o Timing: Skill assessments should be performed until a resident can demonstrate
competency. The sustained level of competency can be measured if a program is
worried about “drift” from desired performance. This would require a recheck of
the skill at some systematic interval.
o Who Performs: Skill assessments should be done through observation of the
actual performance. It is possible that a faculty member could assess through
video review of the performance, but the assessment reflects the skill of the
resident on the performance date.
o Format: A checklist is the most appropriate format, but the checklist may have
some gradation reflecting the quality with which the specific step was performed,
e.g.: not indicated (n/a), not performed but indicated, performed poorly, or
performed well. General scales (such as for O‐SATS above) also exist and
facilitate comparability across specific procedures. Written comments may be
especially helpful for giving feedback.
o Scoring Criteria and Training: There should be guidelines for the checklist
describing the environment for the assessment and a description that
accompanies what is meant by each step on the checklist. For example, if the
checklist includes “washes hands”, does that mean a resident running his/her
hands under water without soap is acceptable? Is scrubbing involved? For a
minimum length of time? It is advisable to indicate to learners and evaluators the
acceptable standard for checklist items. The training could be by reviewing the
written guide. The checklist should contain a written standard by which the
resident would know that the performance demonstrated competency.
Generally, this would mean achieving 100% of the checklist items and/or overall
judgment of competency by the assessor.
o Documentation: At minimum, twice annually as part of semi‐annual review
meetings.
Workflow Procedures
A systematic approach is recommended to maximize the use of the focused assessments
and facilitate data management. A sample workflow document for focused assessment
of surgical skills follows.
References
1. Winckel CP, Reznick RK, Cohen R, Taylor B. Reliability and construct validity of
a structured technical skills assessment form. Am J Surg1994;167(4):423‐7.
2. Reznick R, Regehr G, MacRae H, Martin J, McCulloch W. Testing technical skill
via an innovative ʺbench stationʺ examination. Am J Surg 1997;173(3):226‐30.
18
UCSF GME ‐ 7/08

ANNUALLY
SAMPLE WORKFLOW FOR FOCUSED ASSESSMENT OF SURGICAL SKILLS
TASK BY WHOM WHEN
Review/update evaluation cards: determine Program Director, May
which procedures for which years
Manager
Order evaluation cards Site Coordinator May
Send instructions to each class of residents
and supervising faculty in the OR on how to
use the evaluation cards
Program Director June
Distribute evaluation cards to residents for Program Directors End of year class
their two (2) respective index cases
meetings (May/June)
WITH EACH INDEXED CASE
TASK BY WHOM WHEN
Hand supervising faculty in OR the card for Resident Preferably immediately
the index case
afterward if necessary
Complete evaluation card, give card to Faculty
Immediately after the case
resident and give resident verbal feedback supervising
resident in the OR
Turn in cards to MZ AAIII, SFGH
Resident Within one week of the
coordinator, or UCSF coordinator
procedure
Enter data into a database Site Coordinators Monthly and must be
current when semi‐annual
assessment meetings are
scheduled
AS NEEDED
TASK BY WHOM WHEN
Review performance of residents Program Directors,
Advisors and
Residents
Review process for improvement Program Director,
Manager, Site
Coordinators
19
Periodically and prior to
semi‐annual assessment
meetings
As needed
UCSF GME ‐ 7/08

GME Evaluation Task Force Recommendation:
PROFESSIONALISM
PROFESSIONALISM, INTERPERSONAL AND COMMUNICATION SKILLS
Overview
360 evaluations include observations by a variety of individuals from the multiple
contexts where professionalism and communication skills are demonstrated. Ratings by
self, peers, colleagues and faculty are generally collected using different tools than
ratings by patients. The 360 method captures information on most of the competencies
listed by the ACGME under “Professionalism” and “Interpersonal & Communication
Skills and is highly recommended for every program’ assessment system.
Core Measure for UCSF GME
For Health Care Team and Self‐Evaluation, we recommend the Professional Associate
Evaluation Form, developed by the CREOG (Ob/Gyn) Competency Task Force
For Patient Surveys, we recommend the American Board of Internal Medicine (ABIM)
Patient Survey
Health Care Team Evaluations: PROFESSIONALISM
_______________________
Reliability and Validity:
The optimal number and frequency 360 assessments is uncertain. A feasible minimum
to achieve inter‐rater reliability = .80 may be 5 non‐clinical and 6 clinical raters each on 2
occasions (1).
Preliminary evidence of construct validity shows modest growth in 360 scores
comparing senior vs. junior residents with a magnitude similar to the growth in other
competencies including critical self‐reflection skills (2).
Content validity exists to the extent that the survey items actually assess the
professionalism and communication skills they are intended to measure. The 9‐item
evaluation includes communication (patients/families, nursing/allied staff), respect
(patients, nursing/allied staff), compassion, reliability, honesty/integrity, responsibility,
and advocacy.
20
UCSF GME ‐ 7/08

Administration
o Timing: More frequent observations tied to specific learning experiences provide
more valid data than the minimum, i.e., global impressions provided twice
annually.
o Who Performs: Many individuals can legitimately contribute to 360 assessments.
The list usually includes faculty, residents (supervisors, peers and juniors),
nursing and other clinical staff, and consultants. Selection should be based upon
the individual’s opportunities to directly observe the resident interacting with
patients and the health care team.
N.B. Faculty need to complete the 360 evaluation separately from their global
ratings at the end of the rotation so the data can be summarized and reported
accurately.
o Format: 9 items sample aspects of professionalism and communication skills.
Each item is scored on 9‐point scales with 1‐3 = unsatisfactory, 4‐6 = meets
expectations, and 7‐9=excellent.
o Scoring Criteria and Training: Each item defines a specific trait in clear language.
Standard‐setting would be helpful for consistently distinguishing the 3 levels of
unsatisfactory, the 3 levels of satisfactory and the 3 levels of excellent.
o Documentation: Assessment can be documented on E‐value and learners can
access the results confidentially.
Uses of the Data
o Summarizing the data: Score averages, ranges, and comparative data for the PGY
year are provided as part of the data report residents review before their semi‐
annual meetings. The other data sources for the 360 assessment include the
relevant items (e.g., respect) from the residents’ clinical educator evaluations and
patient surveys. These are also summarized as means and ranges compared with
averages for the PGY year.
o Formative Uses: Assessment results support development of professionalism and
communication skills by individual residents, identify trends in performance
across PGY groups and spur possible improvements to the curriculum in these
two competencies
o Summative Decisions and Remediation: Scores in the unsatisfactory (scores 1‐3)
would trigger remediation and a low satisfactory score (4) would trigger
suggestions for improvement
Workflow Procedures
A systematic approach is recommended to maximize the use of the assessments and
facilitate data management. An example follows in the Appendix.
21
UCSF GME ‐ 7/08

References
1. Murphy DJ, Bruce DA, Mercer SW, Eva KW. The reliability of workplace‐based assessment in
postgraduate medical education and training: a national evaluation in general practice in the
United Kingdom. Adv in Health Sci Educ 2008 DOI 10.1007/s10459‐008‐9104‐8.
2. Learman LA, Autry AM, OʹSullivan P. Reliability and validity of reflection exercises for
obstetrics and gynecology residents. Am J Obstet Gynecol 2008;198(4):461.e1‐8; discussion
461.e8‐10.
22
UCSF GME ‐ 7/08

University of California, San Francisco
Core Measure for UCSF GME: Professionalism / Communication
Subject:
Evaluator:
Site:
Period:
Dates of Activity:
Activity: Health Care Team Evaluation
Evaluation Type: [Evaluator Identity]
Clinical Setting (Question 1 of 13 - Mandatory)
Please indicate the clinical setting where you have interacted with the resident:
Inpatient Wards
ICU
ER
Outpatient Clinic
Continuity Clinic
Other:
Clinical Observations (Question 2 of 13 - Mandatory)
On average how many clinical observations did you have of the resident?
N/A
< 4
5-10
11-20
21 >
Communication: Patients and Families (Question 4 of 13 - Mandatory)
Communicates clearly; is willing to answer questions and provide explanations; willing to listed to patients
and families
Unable to
Assess
Unsatisfactory Satisfactory Excellent
0 1 2 3 4 5 6 7 8 9
Communication: Nursing and Allied Health Staff (Question 5 of 13 - Mandatory)
Consistently demonstrates willingness to listen to nursing and allied health staff
Unable to
Assess
Unsatisfactory Satisfactory Excellent
0 1 2 3 4 5 6 7 8 9
Respectfulness: Patients (Question 6 of 13 - Mandatory)
23
UCSF GME ‐ 7/08

Treats others with respect; does not demean or make others feel inferior; provides equitable care to
patients; uses respectful language when discussing patients; is sensitive to cultural needs of patients
Unable to
Assess
Unsatisfactory Satisfactory Excellent
0 1 2 3 4 5 6 7 8 9
Respectfulness: Nursing and Allied Health Staff (Question 7 of 13 - Mandatory)
Consistently courteous and receptive to nursing and allied health staff; acknowledges and respects roles
of other health care professionals in patient care
Unable to
Assess
Unsatisfactory Satisfactory Excellent
0 1 2 3 4 5 6 7 8 9
Compassion: (Question 8 of 13 - Mandatory)
Is kind to patient and families; appreciates patients and families special needs and accepts
inconvenience when necessary to meet the needs of the patient; consistently attentive to details of
patient comfort
Unable to
Assess
Unsatisfactory Satisfactory Excellent
0 1 2 3 4 5 6 7 8 9
Reliability: (Question 9 of 13 - Mandatory)
Completes and fulfills responsibilities; responds promptly when on call or when paged; assists and fills in
for others when needed
Unable to
Assess
Unsatisfactory Satisfactory Excellent
0 1 2 3 4 5 6 7 8 9
Honesty/Integrity: (Question 10 of 13 - Mandatory)
Knows limits of ability and asks for help when appropriate; is honest and trustworthy; does not falsify
information; committed to ethical principles
Unable to
Assess
Unsatisfactory Satisfactory Excellent
0 1 2 3 4 5 6 7 8 9
Responsibility: (Question 11 of 13 - Mandatory)
Accepts responsibility (does not blame others or the system); committed to self-assessment; responds to
feedback; committed to excellence and self-learning
Unable to
Assess
Unsatisfactory Satisfactory Excellent
0 1 2 3 4 5 6 7 8 9
Advocate: (Question 12 of 13 - Mandatory)
An advocate for patient needs, effectively accesses and coordinates medical system resources to
optimize patient care, seeks to find and correct system causes of medical error
Unable to
Assess
Unsatisfactory Satisfactory Excellent
0 1 2 3 4 5 6 7 8 9
24
UCSF GME ‐ 7/08

Comments: (Question 13 of 13)
Please provide comments regarding resident's strengths and / or areas of needed improvement:
25
UCSF GME ‐ 7/08

ANNUALLY
SAMPLE WORKFLOW FOR HEALTHCARE TEAM EVALUATION
TASK BY WHOM WHEN
Review/update survey form Program Director,
Manager and E‐
Value Administrator
Review/update rotation list Program Manager
and E‐Value
Administrator
Review/update Professional Associates list by
rotation and residents’ peer list
AT THE END OF EACH ROTATION
26
E‐Value
Administrator
May/June
May/June
June/July
TASK BY WHOM WHEN
Schedule professional associate evaluations of E‐Value
by July 1st
residents on E‐Value for intramural rotations Administrator
Solicit input from other professional associates Designated
Within one week of the
involved in resident’s training on that rotation professional
associate
completion of each rotation
Enter resident evaluations into E‐Value Designated
Within two weeks of the
professional
associate and
residents on same
teams
completion of each rotation
Automatic reminders to faculty with outstanding E‐Value software Sent every 1 week for 4
evaluations
weeks
Follow‐up reminder for delinquent faculty and E‐Value
End of the next rotation after
resident evaluations from previous rotation (cc: Administrator the one the evaluation was
Director and Associate Director)
assigned
AS NEEDED
TASK BY WHOM WHEN
Review performance of residents Program Directors,
Advisors and
Residents
Remove and track suspended evaluations E‐Value
Administrator
Update professional associates and their contact
information
E‐Value
Administrator
Periodically and prior to
semi‐annual assessment
meetings
Quarterly
As needed
UCSF GME ‐ 7/08

Patient Surveys: PROFESSIONALISM
______
Reliability and Validity:
The optimal number of patient surveys is uncertain. As initially used in an ABIM
continuing professional development context, 20 surveys were recommended. A
feasibility study with Canadian internal medicine residents showed 12 outpatient
surveys to be associated with poor reliability (0.56) (1).
Data using the Consultation and Relational Empathy (CARE) survey, which is used for
physician accreditation in Scotland, suggest that more than 40 patients may be necessary
for good inter‐rater reliability (.80) with 25 patients providing adequate reliability (.70)
(2).
Content validity exists to the extent that a measure actually assesses the communication
and professionalism skills it is intended to measure. The ABIM’s 10 items sample
multiple aspects of doctor‐patient communication (greeting, listening, establishing
rapport, explaining, inviting participation in decision‐making) and professionalism
(truthfulness, respect, sensitivity to linguistic barriers). The physician characteristics
evaluated using the CARE survey are similar to those assessed using the ABIM survey.
Although the CARE survey has some advantages in how it describes the characteristics
being evaluated, we recommend the ABIM survey because it is so widely used and
studied in the United States.
Administration
o Timing: Patient satisfaction surveys may be obtained regularly as a quality
measure. Otherwise, administration twice annually is the minimum
recommended for assessment of competency.
o Who Performs: Patients under the direct care of the resident.
o Format: The ABIM Patient Survey includes 10 complex items. Both sample
similar aspects of communication and professionalism skills.
o Scoring Criteria and Training: The ABIM survey uses scales ranging from 1‐5
(poor, fair, good, very good, and excellent), uses simple language, and relies upon
the subjective experience of individual patients. Patients receive a general
orientation but no specific instructions regarding the scoring criteria. Although a
9‐point scale is used for the 360’s Health Care Team and Self‐Evaluations, the
patient surveys use only 5 points because finer distinctions are challenging for
patients. This 5‐ vs. 9‐point scale difference is important to note when scores are
summarized and discussed with residents.
27
UCSF GME ‐ 7/08

o Documentation: Summaries with comparative data are made available for review
at the semi‐annual meetings.
Uses of the Data
o Summarizing the data: Score averages, ranges, and comparative data for the PGY
year are provided as part of the data report residents review before their semi‐
annual meetings. The other data sources for the 360 assessment include the
relevant items (e.g., respect) from the residents’ clinical educator evaluations and
patient surveys. These are also summarized as means and ranges compared with
averages for the PGY year.
o Formative Uses: There are many useful ways to use the assessment results to
support development of professionalism and communication skills by individual
residents, identify trends in performance across PGY groups and spur possible
improvements to the curriculum.
o Summative Decisions and Remediation: It is important to set criteria that would
trigger a plan of improvement to explain the criteria to the residents before they
are assessed.
Workflow Procedures
A systematic approach is recommended to maximize the use of the assessments and
facilitate data management. An example follows in the Appendix.
References
1. Tamblyn R, Benaroya S, Snell L, McLeod P, Schnarch B, Abrahamowicz M. The
feasibility and value of using patient satisfaction ratings to evaluate internal medicine
residents. J Gen Intern Med 1994;9(3):146‐52.
2. Murphy DJ, Bruce DA, Mercer SW, Eva KW. The reliability of workplace‐based assessment in
postgraduate medical education and training: a national evaluation in general practice in the
United Kingdom. Adv in Health Sci Educ 2008 DOI 10.1007/s10459‐008‐9104‐8.
.
28
UCSF GME ‐ 7/08

Doctor's Name__________________________
Date_____________
Inpatient Outpatient
(circle one)
ABIM Patient Survey
29
Rating Scale
HOW IS THIS DOCTOR AT. . . Poor Fair Good Very
Good
Greeting you warmly; calling you by the name you
prefer; being friendly, never crabby or rude
Letting you tell your story while listening carefully;
asking thoughtful questions; not interrupting you
while you are talking
Showing interest in you as a person; not acting
bored or ignoring what you have to say
Treating you like you're on the same level; never
"talking down" to you or treating you like a child
Informing you during the physical exam about what
he/she is going to do and why; telling you what
he/she finds
Explaining what you need to know about your
problems, how and why they occurred, and what to
expect next
Using words you can understand when explaining
your problems and treatment; explaining any
technical medical terms in plain language
Discussing options with you and asking your
opinion; offering choices and letting you help
decide what to do; asking what you think before
telling you what to do
Encouraging you to ask questions; answering them
clearly; never avoiding your questions or lecturing
you
Telling you everything; being truthful, upfront and
frank; not keeping things from you that you should
know
Excellent Unable to
Evaluate
1 2 3 4 5 #
1 2 3 4 5 #
1 2 3 4 5 #
1 2 3 4 5 #
1 2 3 4 5 #
1 2 3 4 5 #
1 2 3 4 5 #
1 2 3 4 5 #
1 2 3 4 5 #
1 2 3 4 5 #

ANNUALLY
SAMPLE WORKFLOW FOR PATIENT SATISFACTION QUESTIONNAIRE
TASK BY WHOM WHEN
Review/update survey form Program Director,
Manager, Site
Coordinators
Translate form into Spanish and Chinese and SFGH Coordinator;
estimate numbers of each needed for each clinic Directors, Manager
Order scannable forms Program Director,
Manager, Site
Coordinators
Determine months for survey administration and
inform clinic staff and residents
EVALUATION MONTH
Program Director,
Manager, Site
Coordinators
31
May
May
May
June/July
TASK BY WHOM WHEN
Send a reminder to the continuity clinic staff
(SFGH & MZ); surgical coordinator at SFGH; and
nurses in the gyn clinic at MZ that patient
satisfaction surveys will be administered next
month
Site Coordinators Send one week prior to the
beginning of the survey
months (September and
March)
Distribute forms to clinics Site Coordinators Within one week prior to
the beginning of the survey
months (September and
March)
Present forms to all patients in continuity clinics,
post‐operative visits (SFGH) and gyn clinic (MZ)
Create and place box for surveys at checkout
counters at continuity clinics, post‐operative visits
(SFGH) and gyn clinic ((MZ)
Medical Assistants in
continuity clinics;
surgical coordinator
at SFGH; and nurses
in the MZ gyn clinic
Site Coordinators
with nursing staff
During the two survey
months
(September and March)
First day of each survey
month
(September and March)
Collect forms from boxes Site Coordinators Weekly during the month
and follow‐up for stragglers
the following month
Scan or enter surveys into database E‐Value
Administrator
AS NEEDED
TASK BY WHOM WHEN
Review performance of residents Program Directors,
Advisors and
Residents
Determine response rate and review process for
improvement
Program Director,
Manager, Site
Coordinators
By the 15 th day of the month
following survey month
Periodically and prior to
semi‐annual assessment
meetings
After data is entered

GME Evaluation Task Force Recommendation
PRACTICE BASED LEARNING AND IMPROVEMENT
PBLI: CRITICAL APPRAISAL SKILLS
Overview
Critical appraisal skills are essential for finding, evaluating and translating evidence from
the literature into clinical practice.
Core Measure for UCSF GME
The Critical Appraisal Exercise is to be used to evaluate residents’ ability to answer a
focused clinical question by searching the medical literature, summarizing their findings,
evaluate the limitations of the evidence, and discuss how their practice will change based on
the evidence gathered. It is intended to be used in journal clubs, other critical appraisal
presentations, and after ad hoc searches of the literature to guide clinical management.
Templates for critically appraising articles of specific types can be found at Oxford’s Centre
for Evidence‐based Medicine: http://www.cebm.net/index.aspx?o=1157.
However, these worksheets are directed toward evaluating the quality of a study and not
the critical appraisal abilities of the learner.
Reliability and Validity
The number of evaluators and assessment opportunities to assure adequate reliability are
unknown. Content validity can be established to the extent that key components of critical
appraisal are included in the checklist rating form.
Administration
o Timing: Depends on importance of developing critical appraisal skills. Programs
should assign a certain number that need to be completed satisfactorily in a
specified time frame.
o Who Performs: Designated experts in critical appraisal. Generally, evaluators
would be faculty although experts in library science could evaluate the learners’
search strategies. Learners can also assess their own performance and compare it
to the faculty rating.
o Format: an exercise with instructions completed by the resident followed by a
checklist of 6 components completed by the faculty member
o Scoring Criteria and Training: No faculty training is required. The checklist
components are judged as Yes/No.
o Documentation: At minimum, twice annually as part of semi‐annual review
meetings.
32

Uses of the Evaluation Data
o Formative Feedback: concurrent or same day written evaluation and debriefing
o Tracking Resident Learning: individualized learning plans discussed with a
mentor and skill development over time
o Assess Program Effectiveness: aggregating data across all residents or by PGY
group
o Summative Uses: depending on how important critical appraisal skills are to the
training program, poor performance could trigger remediation and affect
promotion or progress decisions
33

University of California, San Francisco
Core Measure for UCSF GME: Critical Appraisal (PBLI)
Critical Appraisal Exercise (PBLI)
Name of Presenter _____________________________________________
Date _____________________________
Clinical Question(s)
What was learned
Search Strategy and
Search Terms
(MedLine, Cochrane, Textbook,
on-line ref)
*LEVEL OF EVIDENCE
GOOD Large randomized trials with clear-cut results (and low risk of error)
FAIR Small randomized trials with uncertain results (and moderate to high risk of error)
or nonrandomized trials with concurrent or contemporaneous controls
POOR Nonrandomized trials with historical controls or case series with no controls
34
Magnitude of
Expected Effect
(e.g. # needed to treat
for benefit & harm)
Do you plan to change your
practice? How?
Modified from the
CREOG Competency
Task Force
Level of
Evidence*
1 - Refined question to be clinically focused and relevant
Goal Met
� Yes � No
2 - Used logical, focused search strategy � Yes � No
3 - Summarized study design and findings into clinically relevant metric (e.g. NNT) � Yes � No
4 - Critically appraised the study(ies) and identified strengths and threats to validity � Yes � No
5 - Discussed applicability of study findings to patient population or context at hand � Yes � No
6 - Considered health policy implications of findings (e.g. feasibility, cost, harms) � Yes � No
7 - Discussed limitations of current evidence � Yes � No
8 - Discussed areas of future research
Signature
� Yes � No
Evaluator ____________________________________ _______________________________
Please return this form to either the SFGH, UCSF or MZ Residency Program Coordinators

GME Evaluation Task Force Recommendation
PRACTICE BASED LEARNING AND IMPROVEMENT
PBLI: TEACHING SKILLS
Overview
Per ACGME Common Program Requirements teaching skills are a component of PBLI
(“participate in the education of patients, families, students, residents and other health
professionals”) and ICS (“communicate effectively with physicians, other health
professionals, and health related agencies”).
If respect for learners or patients is included, teaching evaluations also provide evidence of a
component of Professionalism: “compassion, integrity and respect for others”
Core Measure for UCSF GME
Observation is the primary method by which clinical educators are rated. The SOM Clinical
Educator Teaching Evaluation Form was developed at UCSF as a global assessment
conducted at the end of a clinical rotation to assess the quality of medical student teaching
by residents and faculty. This form consists of 19 items: 11 items on a 5‐point Likert‐type
scale, 4 are narrative/open ended items, and 4 items are triggered only if low scores are
received on certain critical items on the form. However, experience and internal studies of
the form indicate it can be shortened without losing any reliability. We recommend this
shorter form as our core measure of clinical teaching.
Reliability and Validity
There are many instruments developed to measure clinical teaching effectiveness. Most of
these instruments do tend to measure a global teaching effectiveness score, interpersonal,
and clinical teaching/pedagogic practices dimensions, and have high internal consistency.
Each item in the SOM Clinical Educator form includes detailed anchors illustrating each
point on the 5‐point scale (1=poor, 5=excellent). Due to the internal consistency of these
forms, a shorter item set has adequate reliability and content validity. Our proposed core
measure includes: conveyance of information, teaching enthusiasm, direction and feedback,
promotion of critical thinking, treat me with respect, treat others with respect, and overall
teaching effectiveness. Research has recommended that scales be tailored to learner
(medical student and resident) and setting (e.g. outpatient vs. inpatient); hence, additional
items may be included but should be similar in format to the other items and include clear
anchors.
Current Procedures by the SOM
o Administration: Practices and evaluation frequency for faculty/residents as clinical
teachers vary by each (SOM) clerkship. Some rotations require certain
number/types of interactions to occur in order for form to be assigned, some ask
35

learners to designate faculty/residents they worked with and then the form is
assigned, so on and so forth.
o Dissemination: The forms are disseminated by each department however the
(SOM) Office of Medical Education centrally oversees policies surrounding the
used of a standardize form, user management, reporting, and procedural record
keeping.
o Reporting: Faculty and residents are able to view their own evaluation in real time.
The SOM Office of Medical Education annually reports on aggregate faculty and
resident teaching scores for each clerkship by site.
Administration by GME programs
o Frequency: It is recommended that teaching be evaluated after a designated
number of interactions between teacher and learner. The number of interactions is
dependent on the length of the rotation and should be designated accordingly.
o Who Performs: The learners (students, more junior residents).
o Scoring Criteria and Training: It is recommended that the form be publicly visible
and that evaluators know the scoring/rating in advance so that they know what
they are rating about their instructors. There is no training associated with the use
of this evaluation.
o Documentation: Twice annually as part of semi‐annual review meetings.
Use of Data
How assessment results are used is dependent on the program. Timely feedback both
written and oral between teachers and program directors will help encourage those good
teachers and well as remediate and improve teaching. It is recommended that certain
critical items on the form (e.g. teaching effectiveness, respect) create low score triggers.
These trigger should prompt additional evaluation items, closed ended or narrative, in order
to allow the evaluator to elaborate on the low scores. A low score on any of the items,
particularly critical items, should trigger remediation.
o Formative uses: Most important use as part of mentored review of progress,
guiding individualized learning plans
o Summative uses: Usually not unless low scores contribute to a pattern of difficulty
in one or more competency areas
o Program benchmarking: Yes ‐ as % of residents and faculty achieving a criterion
goal or standard for direct teaching performance
Optional Items
The GME Evaluation TF recommends that all GME programs use the SOM short form as a
core set to facilitate benchmarking for individual programs and the School. Review of the
other evaluation tools revealed potentially useful items that programs may choose to add to
the basic form. We have included these as “Item Bank Recommendations.” Remember
ultimately what matters the most is the overall teaching effectiveness and comments.
36

Item Bank for Optional Use:
1. During this time I personally interacted with or observed the resident and base this
evaluation on (very concrete item based on hours of contact)
2. Refers learner to pertinent references for further reading
3. Reviews exam findings with learner
4. Discusses differential diagnosis and work‐up with learner
5. Reviews treatment options with learner
6. Provides follow‐up to learners on interesting cases
7. Takes time to stress proper surgical technique
8. Discusses rational for surgical judgment
9. Please rate how well this resident emphasized problem solving (i.e. thought process
leading to decisions)
Other Notes:
o Neurological Surgery had each of their questions categorized by ACGME
competencies. This was nice – easy to track later.
o The IM Cardiology UC Consult form was nicely tailored to the specialty and type
of education.
o The LEAH Fellowship form was nice and brief although we would recommend a
five point scale and spell out teaching effectiveness.
Specific Assessment of Lecturing Skills
Programs may choose to assess the development of residents’ didactic teaching skills, either
alone or in conjunction with assessment of critical appraisal skills (e.g., after a lecture
reviewing the evidence on a specific clinical question). The Teaching Observation Form
developed by the Academy of Medical Educators is an excellent source of tailored and
structured feedback by a trained peer or faculty evaluator. The resident giving the lecture
would meet in advance with the trained observer and prioritize components of the lecture
for feedback. Following the lecture structured feedback would be shared including future
plans for improvement.
Quantitative measures of lecture quality also exist. An example is provided that asks
students to rate 8 attributes of the instructor on a 5‐level scale from strongly disagree to
strongly agree. This sort of measure may be more appropriate for following the
development of didactic teaching skills than the Clinical Educator Teaching Evaluation
Form, which addresses global teaching performance in the clinical context.
37

University of California, San Francisco
Core Measure for UCSF GME: Clinical Teaching (PBLI)
N.B. This measure comprises a core subset of items included in the SOM Clinical Educator
Teaching Evaluation Form used to assess medical student teaching by residents and faculty.
Evaluation information entered here will be made available to the evaluated person in
anonymous and aggregated form only.
Please rate your instructor's ability to do the following:
Conveyance of Information
Convey information clearly.
Insufficient
contact to judge.
1 Poor communication
skills, conveying information in
unclear manner or consistently
failing to communicate
important points to students.
Teaching Enthusiasm
Provide enthusiastic and stimulating teaching.
Insufficient
contact to judge.
1 Lack of enthusiasm
for teaching students; does
not stimulate students’
interest or curiosity in clinical
setting.
Direction and Feedback
Provide direction and feedback.
Insufficient
contact to judge.
1 Does not define
expectations; fails to
provide student with
direction or feedback about
clinical performance;
devotes little time or
attention to helping
students improve.
Promotion of Critical Thinking
Promote critical thinking.
Insufficient
contact to judge.
1 Does not discuss
clinical reasoning and
knowledge of underlying
mechanisms of disease with
students; does not encourage
use of the literature to improve
patient care or pursue selfdirected
learning.
2 3 Good
communication skills;
usually conveys
information in a clear,
comprehensive manner.
2 3 Usually
enthusiastic about
teaching; maintains an
interest in students’
learning.
2 3 Discusses
expectations; provides
some direction and
feedback about clinical
performance; devotes
adequate time and
attention to helping
students improve.
2 3 Promotes critical
thinking through clinical
reasoning, emphasis on
underlying mechanisms of
disease, and use of the
literature to improve patient
care and encourage selfdirected
learning.
38
4 5 Excellent
communication skills;
consistently conveys
information in exceptionally
clear, comprehensive
manner.
4 5 Consistently
enthusiastic about teaching;
outstanding at stimulating
students’ interest in learning.
4 5 Provides clear
guidelines about expectations;
provides specific, useful
feedback to student verbally
about strengths and areas for
improvement; exceptional level
of time and attention devoted to
helping students improve.
4 5 Exceptional ability to
promote critical thinking through
clinical reasoning, emphasis on
the underlying mechanisms of
disease, and use of the literature
to improve patient care and
encourage self-directed learning.

Treat me with Respect
I was treated with respect by this individual
Insufficient
contact to judge.
1 This individual
consistently failed to treat
me with respect and
generally displayed an
unprofessional or abusive
manner during all
interactions.
2 This individual
treated me with respect
approximately half of the
time; displayed an
unprofessional or
disrespectful manner
during the remainder of the
time.
39
3 This
individual
treated me
with respect
most of the
time.
4 This
individual
treated me
with respect
almost
always.
5 This
attending
consistently
treated me with
respect
throughout the
rotation.
Treat me with Respect - Reasons
If you answered 2 or below on the previous question, please indicate in which way(s) you were not
treated with respect by this educator or resident. (Mandatory for answers of 2 or below on the
previous question.)
Belittled or humiliated me
Spoke sarcastically or insultingly to me
Intentionally neglected or left me out of the communications
Subjected me to offensive sexist remarks or names
Subjected me to racist or ethically offensive remarks or names
Engaged in discomforting humor
Denied me training opportunities because of my gender
Required me to perform personal services (i.e. babysitting, shopping)
Threw instruments/bandages, equipment etc.
Threatened me with physical harm (e.g. hit, slapped, kicked)
Created a hostile environment for learning
Other
Treat me with Respect - Other
If you chose other in the previous question, please explain in the comment section below.
Treat Others with Respect
I observed others (students, residents, staff, patients) being treated with respect by this individual
Insufficient
contact to judge.
1 This individual
consistently failed to treat
2 This individual
treated others with respect
3 This
individual
4 This
individual
5 This
attending

others with respect and
generally displayed an
unprofessional or abusive
manner during all
interactions.
Treat Others with Respect - Reasons
approximately half of the
time; displayed an
unprofessional or
disrespectful manner
during the remainder of the
time.
40
treated others
with respect
most of the
time.
treated others
with respect
almost
always.
consistently
treated others
with respect
throughout the
rotation.
If you answered 2 or below on the previous question, please indicate in which way(s) Patients or
Health Professionals were not treated with respect by this educator or resident. (Mandatory for
answers of 2 or below on the previous question.)
Patients - Discussed confidential information in an inappropriate setting (e.g. cafeteria, elevator)
Patients - Made derogatory or disrespectful comments about a patient or family
Patients - Treated patients differently because of their financial status, ethnic background, religious
preferences or sexual orientation
Patients - Threw instruments/bandages, equipment etc.
Patients - Created a hostile environment for patient care and/or learning
Health Professionals - Made derogatory or disrespectful comments about some health professionals
Health Professionals - Treated health professionals differently because of their financial status, ethnic
background, religious preferences or sexual orientation
Health Professionals - Made offensive sexist, racist, or ethnically insensitive remarks/names about
some health professionals
Other
Treat Others with Respect - Other
If you chose other in the previous question, please explain in the comment section below.
Teaching Skills, Overall
Overall teaching effectiveness.

Insufficient
contact to judge.
1 This attending was an overall
poor teacher, either due to inadequate
time spent teaching medical students,
ineffective style, or unprofessional
manner .
2 3 This attending was an
overall good teacher through
dedication of adequate time to
teaching and a generally
effective style.
41
4 5 This attending
was an overall
excellent teacher
through dedication of
time to teaching and a
highly effective style,
enabling significant
skill development
throughout the
rotation.
Instructor Strengths
What are the strengths of this instructor? (These comments will be viewed by the instructor, but will be
anonymous and aggregated. For comments to be effective feedback, please be direct, specific, and
constructive. General comments such as “good instructor” are too non-specific to be of value.)
Instructor Improvements
How could this instructor improve? (These comments will be viewed by the instructor, but will be
anonymous and aggregated. For comments to be effective feedback please be direct, specific, and
constructive. General comments such as “bad instructor” are too non-specific to be of value.)
Confidential Comments, Educator
This area is for giving constructive or corrective feedback that you don't feel comfortable giving directly.
These comments are CONFIDENTIAL and will NOT go directly to the educator. They will be forwarded
ANONYMOUSLY to the program director(s). Please be thoughtful, professional, and constructive in your
feedback.
OPTIONAL CONFIDENTIAL COMMENT
If you are willing to be contacted by the clerkship director to address a particularly concerning issue, please
include your name and contact information below. This will only go the clerkship director and/or the site
director with the goal of appropriately addressing the raised concerns.

UCSF Academy of Medical Educators
TOP Observation Form­ Lecture
NAME: _____________________________________ OBSERVER: ___________________________
TOPIC: ______________________________________________________________________________
FOCUS OF OBSERVATION (discuss w/ faculty in advance):
INTRODUCTION
1. Introduced topic, stated objectives, offered
preview.
2. Gained attention and motivated learning.
3. Established climate for learning and for
participation.
42
OBSERVATIONS
BODY OF LECTURE OBSERVATIONS
4. Presented 3 – 5 main points in clear and
organized fashion.
5. Provided supporting materials, examples, and
summaries.
6. Content level
7. Effectively used visuals, handouts, and/or
demonstrations. Include AV problems (if any),
effective use of slides (set stage for each slide,
focused audience on important parts of slides),
use of pointer.
8. Varied presentations (Used blackboard, slides,
visuals).
9. Transitions between topics.
CONCLUSION OBSERVATIONS
10. Summarized major principles, key points
without introducing new materials.
11. Provided closure or stimulated further thought.

TEACHER DYNAMICS OBSERVATIONS
12. Exhibited enthusiasm and stimulated interest in
content.
13. Used appropriate voice, gestures, movement,
and eye contact. Avoidance of unconscious use
of repeated words (eg “um”, “ok”).
14. Encourage active participation.
15. Used questions to stimulate thought and
discussion. Response to questions (repeated or
rephrased question, concise answer).
DEBRIEF
1. ELICIT SELF-ASSESSMENT BY MENTEE FIRST.
2. SUMMARIZE YOUR ASSESSMENT OF MENTEE’S STRENGTHS AND YOUR RECOMMENDATIONS
(KEEP IN MIND AREAS OF FOCUS).
STRENGTHS
1.
2.
3.
3. ACTION PLAN (RESIDENT TO COMPLETE)
RECOMMENDATIONS
43

Instructor Student Rating Form
Instructions: Please fill in the bubble that best describes your rating using the following scale:
1 = Strongly Disagree
2 = Disagree
3 = Unsure
4 = Agree
5 = Strongly Agree
Core Items 1 2 3 4 5
1. Organization: Instructor presented material
systematically and sequentially.
2. Clarity: Instructor communicated effectively, presented
content clearly, and gave understandable responses to
questions.
3. Enthusiasm: Instructor effectively stimulated learner
interest.
4. Contributions: Instructor discussed recent developments
in the field, directed students to current reference materials,
and provided additional materials to cover current topics.
5. Rapport: Instructor listened attentively, was interested in
students’ progress, and provided constructive feedback.
6. Professional Characteristics: Instructor demonstrated
respect for students, cultural awareness, respect for health
professions, and other aspects of professionalism.
7. Attitude: Instructor was concerned about students
learning the material, encouraged class participation, and
respected differing views.
8. Overall: I would rate this instructor as excellent. O O O O O
Revised 6/10/05
44
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O

ADDITIONAL RECOMMENDATIONS
OF THE
EVALUATION TASK FORCE
45

GME Evaluation Task Force Recommendation
MEDICAL KNOWLEDGE
Overview
Multiple choice examinations (MCQ) of medical knowledge are available from specialty
boards, professional societies and program director organizations. In many cases they are
required as in‐service examinations to track residents’ progress and set the stage for written
board certification examinations. Although MCQ’s provide valid assessments of medical
knowledge, oral examinations and presentations of clinical cases are considered better
measures of clinical reasoning.
Reliability and Validity
Generally MCQ’s are developed to achieve high internal consistency reliability for their
subscales. Content validity is maximized by assuring that the examination items sample the
full range of core knowledge. Construct validity is demonstrated when higher level
residents have a greater % correct than lower level residents. Although performance on one
MCQ tends to predict performance on subsequent ones, evidence is mixed correlating MCQ
performance with other aspects of medical knowledge such as clinical reasoning.
Administration
o Timing: usually once per year
o Who Performs: generally a secure examination administered by staff according to
guidelines of the in‐service examination
o Format: Each item contains an introductory statement or ‘stem’ followed by four or five
response options, only one of which is correct. The stem is usually a patient case, clinical
findings, or displays data graphically. A typical half‐day examination has 175 to 250 test
questions.
o Scoring Criteria and Training: Completed exams are generally returned to the
organization that provides the test for scoring. Score reports can include raw % correct,
scores standardized for PGY level, and subscores in key content areas.
o Documentation: Achievement of the medical knowledge competency must occur at least
twice a year at the semi‐annual review meeting. MCQ performance can inform one of the
meetings, and other knowledge assessments (global evaluations, assessments of clinical
reasoning, progress toward reading goals from last year’s MCQ) can inform both meetings.
Uses of the Data
o Comparing the test scores on in‐training examinations with national statistics can serve to
identify strengths and limitations of individual residents to help them improve.
o Summative Decisions: MCQ performance falling short of a minimum passing threshold
could delay or prevent a resident from advancing or graduating. Generally, however, such
46

decisions should be based on overall assessments of medical knowledge including clinical
reasoning.
o Remediation Threshold: Programs should communicate what performance on the MCQ
would require remediation. The threshold for remediation may be determined by a national
or local standard for passing performance or a score that portends difficulty passing the
board certification examination. Generally a specific program of study would be established
to close gaps in knowledge, and progress would be assessed short‐term using written or oral
examinations. [N.B. because in‐service examinations are often administered only once
annually, programs may need to rely on other measures of progress].
Comparing test results aggregated for residents in each year of a program can be helpful to
identify residency training experiences that might be improved.
47

GME Evaluation Task Force Recommendation
SYSTEMS‐BASED PRACTICE
The evaluation of systems‐based practice poses unique challenges because of the variation in
learning and assessments opportunities among GME programs. We recommend that
programs first identify where residents are routinely called upon to demonstrate systems
based practice skills and then select methods for evaluating and documenting these skills.
We list examples of these activities and how they might be assessed.
Morbidity and Mortality Conference Presentation
These occur on a regular basis in many residency programs and include resident
presentations of clinical cases with an assessment of how care could be improved. If this
assessment routinely includes an analysis of the system’s role in prevention of medical errors
and harms, the resident’s analysis can be evaluated using a form or checklist.
Quality Improvement Meeting
Resident involvement in quality improvement activities is an ACGME requirement. If a
resident routinely participates in a clinical quality improvement meeting during a particular
PGY year or rotation, their participation can be assessed by peers or supervisors on the
committee.
Cost‐Effective Care Practices
These skills are demonstrated when residents make decisions about screening tests,
diagnostic tests treatments, and site of care decisions. If residents routinely present their care
plans to faculty in rounds, morning report, conferences (pre‐op, tumor board) or other
venues, the faculty can evaluate the degree to which the resident demonstrates an awareness
of cost‐effectiveness.
Quality Improvement Project
There are many venues where individual residents or groups of residents can identify a
process or outcome they wish to improve, develop a plan that includes consideration of
system issues, implement the plan and measure its success. The full project or any of its
components could be assessed by experts in quality improvement. The Quality Improvement
Project Assessment Tool (QIPAT‐7) is provided below as on example of how a resident
project can be assessed. The QIPAT‐7 was developed based upon the input of national QI
experts and its reliability has been demonstrated in an evaluation of 45 resident QI proposals
(Leenstra 2007).
48

The HealthCare Matrix from Vanderbilt uses quality aims from the Institute of Medicine and
the ACGME competencies together to assess and improve care. The Matrix is described
further in this abstract from the ACGME eBulletin from December 2006, page 10‐11 (Bingham
2005).
Using a Healthcare Matrix to Assess Care in Terms of the IOM Aims and the ACGME
Competencies
Doris Quinn PhD, John Bingham MHA, Vanderbilt University Medical Center
The study assessed how residents and faculty are using the HealthCare Matrix to assess and
improve care. Whether care is safe, timely, effective, efficient, equitable, or patient‐centered is
juxtaposed against the ACGME competencies. When care is assessed in this manner, learning
the competencies becomes very relevant to the outcomes of care. It presented the work of
internal medicine residents who on their Ambulatory Rotation: 1) utilized the Matrix to assess
the care of their patients; 2) demonstrated use of QI tools to improve care; and 3) improved
publicly reported metrics for AMI and CHF by focusing in particular, system‐based practice
and practice‐based learning and improvement. Residents first utilize the Matrix to assess care
of one of their patient’s. Then, as a group, they choose a publicly reported metric and complete
matrices for a panel of patients. The data from the matrices informs residents as to where more
information or improvement is needed. This becomes the basis for an improvement project
which is ultimately presented to senior leaders. To date, residents have improved the care of
patients with pneumonia, coronary artery disease, diabetes, and processes including obtaining
consults, the VA phone Rx system and others. Public metrics of quality from CMS, JCAHO,
and Leapfrog are utilized in the assessment. When the ACGME competencies are combined
with the IOM aims and used to assess and improve care of patients in “real time”, developing
the competencies becomes “the way residents learn” and not a burden or “add on”. This
process allows residents, who are the most knowledgeable about workarounds and flaws in
the system, to use their experience to improve care. Residents, faculty, the
institution, and most importantly, the patients benefit.
References
Bingham JW, Quinn DC, Richardson MG, Miles PV, Gabbe SG. Using a healthcare matrix to
assess patient care in terms of aims for improvement and ACGME core competencies. JC
Journal on Quality and Patient Safety 2005;32(2): 98–105.
Leenstra JL, Beckman TJ, Reed DA, Mundell WC, Thomas KG, Krajicek BJ, Cha SS,
Kolars JC, McDonald FS. Validation of a method for assessing resident physiciansʹ quality
improvement proposals. J Gen Intern Med. 2007 Sep;22(9):1330‐4. Epub 2007 Jun 30
50

GME Evaluation Task Force Recommendation
PROGRESS REPORT FOR SEMI‐ANNUAL MEETING
The ACGME requires a semi‐annual performance review meeting to discuss each resident’s
progress toward goals and objectives for the year. As programs implement more resident
evaluations, it will be essential to summarize the data on key outcomes and compare
performance to the resident’s goals or benchmarks. The Task Force recommends a progress
report following the template illustrated below. The progress report should be prepared in
advance of the semi‐annual meeting, ideally by having evaluation data electronically
populate the Performance Measures for each resident. Residents would review the report
and come prepared to discuss their progress with the program director or designee. If a
portfolio‐based system is used the resident would also provide evidence of learning they
selected for showcasing.
51

PROGRESS REPORT FOR SEMI‐ANNUAL MEETING
Area / Evaluation Instrument Definition Mean (SD) or
Percent
Evidence of Resident Learning, including ACGME Competencies
Global Evaluations
Medical Knowledge
Patient Care
Professionalism
Interpersonal & CS
Practice‐Based Learning
Systems‐Based Practice
Medical Knowledge: MCQ
Overall Score
Subscore A
Subscore B
Subscore C.
9‐option items completed after
each rotation
Standardized score where 200 is
the mean and 20 is the SD based
on national norms for each PGY
52
6.8 (1.1)
7.1 (0.9)
7.6 (0.8)
8.3 (0.6)
7.0 (1.0)
6.2 (0.5)
220
201
224
215
Standard
(Goal)
Patient Care: Mini‐CEX Average of 7 9‐option items 7.7 (1.2) 5
Patient Care: Focused
Assessment of Skills
Procedures
Communication
Prof / ICS: Team
Percent of checklist items
performed correctly
82%
90%
5
5
5
5
5
5
200
200
200
200
70%
90%
Performance
Relative to
Standard
Average of 9 9‐option items
8.2 (0.7)
5
Met
Prof / ICS: Self
Average of 9 9‐option items
8.2 (0.7)
5
Met
Prof / ICS: Patients Average of 10 5‐option items
4.5 (0.2) 4.0
Met
PBLI – Student teaching Average of 6 5‐option items
4.2 (0.5) 4.0
Met
PBLI – Critical appraisal % checklist items met 90% 80% Met
SBP – M&M evaluation
Case logs
Procedure A
Procedure B
Procedure C
Met
Met
Met
Met
Met
Met
Met
Met
Met
Met
Met
Met
Met
Average of 8 5‐option items 3.8 (1.2) 4.0 Met
Cumulative number of cases
since residency start
Scholarly Project Progress PGY‐specific goals met?
Learning Plan Progress Progress adequate since last
review?
42
105
23
30
60
30
Met
Met
Not Met
Yes Yes Met
Yes Yes Met
Progress report form adapted from Knight DA, Vannatta PM, O’Sullivan PS. A Process to Meet the Challenge
of Program Evaluation and Program Improvement. ACGME Bulletin 2006 (Sept): 5‐8.

GME Evaluation Task Force Recommendation
CLOSING THE LOOP: ANNUAL PROGRAM REVIEW
The ACGME requires programs to conduct a formal and structured evaluation of the
curriculum at least annually. Details of this review are outlined in the ACGME Common
Program Requirements.
The annual review must include consideration of whether the program’s self‐determined
benchmarks have been met in the following areas: resident performance; faculty
development; graduate performance, including performance of program graduates on the
certification examination; and, program quality. Action plans must be created to address
areas in which goals have not been achieved, and the faculty at large must approve the plans
of action. Goals that emerge from the annual program review one year become
benchmarks for the review next year, closing the loop and facilitating the longitudinal and
continuous improvement of program quality.
The Task Force recommends that sufficient time be set aside for a full discussion of the
program’s progress and determination of the goals and action plan for the coming year.
Programs should not underestimate the time and effort required to conduct a systematic
review. Although the ACGME requirements can be met by conducting the annual meeting
of faculty and resident representatives, many programs prefer a dedicated half‐day retreat
for this comprehensive review.
Several challenges emerge for conducting a robust annual program review. Implementing
the Evaluation Task Force recommendations for ACGME competency assessment will yield
resident performance data for each of the evaluation tools. Other required components
include evaluations of faculty teaching and confidential evaluations of the program by
residents and faculty. These data can be summarized in a progress report along with other
program performance measures. The report can compare the program’s outcomes to goals
set from the prior year.
In this section we provide recommendations on assessment of faculty teaching, program
evaluations by resident and faculty, and methods for selecting and reporting data essential
to include in the annual program review.
53

GME Evaluation Task Force Recommendation:
CONFIDENTIAL RESIDENT EVALUATION OF FACULTY TEACHING
Recommended Assessment Tool
Observation is the primary method by which clinical educators are rated. The SOM Clinical
Educator Teaching Evaluation Form was developed at UCSF as a global assessment
conducted at the end of a clinical rotation to assess the quality of medical student teaching
by residents and faculty. This form consists of 19 items: 11 items on a 5‐point Likert‐type
scale, 4 are narrative/open ended items, and 4 items are triggered only if low scores are
received on certain critical items on the form. However, experience and internal studies of
the form indicate it can be shortened without losing any reliability. We recommend this
shorter form as our core measure of clinical teaching.
Reliability and Validity
There are many instruments developed to measure clinical teaching effectiveness. Most of
these instruments do tend to measure a global teaching effectiveness score, interpersonal,
and clinical teaching/pedagogic practices dimensions, and have high internal consistency.
Each item in the SOM Clinical Educator form includes detailed anchors illustrating each
point on the 5‐point scale (1=poor, 5=excellent). Due to the internal consistency of these
forms, a shorter item set has adequate reliability and content validity. Our proposed core
measure includes: conveyance of information, teaching enthusiasm, direction and feedback,
promotion of critical thinking, treat me with respect, treat others with respect, and overall
teaching effectiveness. Research has recommended that scales be tailored to learner
(medical student and resident) and setting (e.g. outpatient vs. inpatient); hence, additional
items may be included but should be similar in format to the other items and include clear
anchors.
Administration by GME programs
o Frequency: It is recommended that faculty teaching be evaluated after a designated
number of interactions between faculty and residents. The number of interactions is
dependent on the length of the rotation and should be designated accordingly.
Immediately after a clinical rotation ends would be optimal, and the minimum
frequency would be twice annually.
o Who Performs: Residents.
o Scoring Criteria and Training: It is recommended that the form be publicly visible
and that evaluators know the scoring/rating in advance so that they know what
they are rating about their instructors. Faculty should also be aware of the criteria
by which their teaching will be judged. There is no training associated with the use
of this evaluation.
o Documentation: Annually as part of the program review meeting and otherwise as
indicated by faculty mentorship meetings, promotion & advancement meetings,
etc.
54

Use of Data
Each department and division will establish its own practices for review of faculty teaching
evaluations. However, the program director is responsible for deciding which faculty
members will teach the residents in the program. The ACGME requires an annual program
review meeting at which time the aggregate data on faculty teaching is reviewed and
compared with the program’s internally defined standards. Action plans are required if
standards are not met.
Optional Items
The GME Evaluation TF recommends that all GME programs use the SOM form as a core
set to facilitate benchmarking for individual programs and the School. Review of the other
evaluation tools revealed potentially useful items that programs may choose to add to the
basic form. We have included these as “Item Bank Recommendations.” Remember
ultimately what matters the most is the overall teaching effectiveness and comments.
Item Bank Recommendations:
1. During this time I personally interacted with or observed the faculty and base this
evaluation on (very concrete item based on hours of contact)
2. Refers resident to pertinent references for further reading
3. Reviews exam findings with resident
4. Discusses differential diagnosis and work‐up with resident
5. Reviews treatment options with resident
6. Provides follow‐up to the residents on interesting cases
7. Takes time to stress proper surgical technique
8. Discusses rational for surgical judgment
9. Please rate how well this attending emphasized problem solving (i.e. thought process
leading to decisions
10. Monitored my stress level (it requires a grammatically different scale than what they
have included – item is not a must but definitely an interesting idea to monitor burnout)
Other Notes:
o Neurological Surgery had each of their questions categorized by ACGME
competencies. This was nice – easy to track later.
o The IM Cardiology UC Consult form was nicely tailored to the specialty and type
of education.
o The LEAH Fellowship form was nice and brief although I would recommend a five
point scale and spell out teaching effectiveness.
55

Faculty Development: The UCSF Academy of Medical Educators uses Teaching
Observation (TOP) Forms to provide structured feedback by a trained peer who has
observed a faculty member doing a lecture or facilitating a small group. The forms include
no scales and are not scored. Their principal use is to facilitate and tailor feedback. The
number of faculty receiving structured feedback on their teaching can be used as a measure
of faculty development for the Annual Program Review.
56

University of California, San Francisco
Confidential Resident Evaluation of Faculty Teaching
N.B. This measure comprises a core subset of items included in the SOM Clinical Educator
Teaching Evaluation Form.
Evaluation information entered here will be made available to the evaluated person in
anonymous and aggregated form only.
Please rate your instructor's ability to do the following:
Conveyance of Information
Convey information clearly.
Insufficient
contact to judge.
1 Poor communication
skills, conveying information in
unclear manner or consistently
failing to communicate
important points to residents.
Teaching Enthusiasm
Provide enthusiastic and stimulating teaching.
Insufficient
contact to judge.
1 Lack of enthusiasm
for teaching residents; does
not stimulate residents’
interest or curiosity in clinical
setting.
Direction and Feedback
Provide direction and feedback.
Insufficient
contact to judge.
1 Does not define
expectations; fails to
provide resident with
direction or feedback
about clinical
performance; devotes little
time or attention to helping
residents improve.
Promotion of Critical Thinking
Promote critical thinking.
Insufficient
contact to judge.
1 Does not discuss
clinical reasoning and
knowledge of underlying
mechanisms of disease
with residents; does not
encourage use of the
literature to improve patient
care or pursue self-directed
learning.
Treat me with Respect
I was treated with respect by this individual
2 3 Good
communication skills;
usually conveys
information in a clear,
comprehensive manner.
2 3 Usually
enthusiastic about
teaching; maintains an
interest in residents’
learning.
2 3 Discusses
expectations; provides
some direction and
feedback about clinical
performance; devotes
adequate time and
attention to helping
residents improve.
2 3 Promotes critical
thinking through clinical
reasoning, emphasis on
underlying mechanisms of
disease, and use of the
literature to improve
patient care and
encourage self-directed
learning.
57
4 5 Excellent
communication skills;
consistently conveys
information in exceptionally
clear, comprehensive
manner.
4 5 Consistently
enthusiastic about teaching;
outstanding at stimulating
residents’ interest in
learning.
4 5 Provides clear
guidelines about expectations;
provides specific, useful
feedback to resident verbally
about strengths and areas for
improvement; exceptional level
of time and attention devoted
to helping residents improve.
4 5 Exceptional ability to
promote critical thinking
through clinical reasoning,
emphasis on the underlying
mechanisms of disease, and
use of the literature to
improve patient care and
encourage self-directed
learning.

Insufficient
contact to judge.
1 This individual
consistently failed to treat
me with respect and
generally displayed an
unprofessional or abusive
manner during all
interactions.
Treat me with Respect - Reasons
2 This individual
treated me with respect
approximately half of the
time; displayed an
unprofessional or
disrespectful manner
during the remainder of
the time.
58
3 This
individual
treated me
with respect
most of the
time.
4 This
individual
treated me
with respect
almost
always.
5 This
attending
consistently
treated me with
respect
throughout the
rotation.
If you answered 2 or below on the previous question, please indicate in which way(s) you
were not treated with respect by this educator or resident. (Mandatory for answers of 2 or
below on the previous question.)
Belittled or humiliated me
Spoke sarcastically or insultingly to me
Intentionally neglected or left me out of the communications
Subjected me to offensive sexist remarks or names
Subjected me to racist or ethically offensive remarks or names
Engaged in discomforting humor
Denied me training opportunities because of my gender
Required me to perform personal services (i.e. babysitting, shopping)
Threw instruments/bandages, equipment etc.
Threatened me with physical harm (e.g. hit, slapped, kicked)
Created a hostile environment for learning
Other
Treat me with Respect - Other
If you chose other in the previous question, please explain in the comment section below.
Treat Others with Respect
I observed others (residents, residents, staff, patients) being treated with respect by this
individual
Insufficient
contact to judge.
1 This individual
consistently failed to treat
others with respect and
generally displayed an
2 This individual
treated others with respect
approximately half of the
time; displayed an
3 This
individual
treated others
with respect
4 This
individual
treated others
with respect
5 This
attending
consistently
treated others

unprofessional or abusive
manner during all
interactions.
unprofessional or
disrespectful manner
during the remainder of the
time.
59
most of the
time.
almost
always.
with respect
throughout the
rotation.
Treat Others with Respect - Reasons
If you answered 2 or below on the previous question, please indicate in which way(s)
Patients or Health Professionals were not treated with respect by this educator or resident.
(Mandatory for answers of 2 or below on the previous question.)
Patients - Discussed confidential information in an inappropriate setting (e.g. cafeteria, elevator)
Patients - Made derogatory or disrespectful comments about a patient or family
Patients - Treated patients differently because of their financial status, ethnic background,
religious preferences or sexual orientation
Patients - Threw instruments/bandages, equipment etc.
Patients - Created a hostile environment for patient care and/or learning
Health Professionals - Made derogatory or disrespectful comments about some health
professionals
Health Professionals - Treated health professionals differently because of their financial status,
ethnic background, religious preferences or sexual orientation
Health Professionals - Made offensive sexist, racist, or ethnically insensitive remarks/names
about some health professionals
Other
Treat Others with Respect - Other
If you chose other in the previous question, please explain in the comment section below.
Teaching Skills, Overall
Overall teaching effectiveness.
Insufficient
contact to judge.
1 This attending was an
overall poor teacher, either
due to inadequate time spent
teaching medical residents,
ineffective style, or
unprofessional manner .
2 3 This attending
was an overall good
teacher through
dedication of adequate
time to teaching and a
generally effective
style.
4 5 This attending was an
overall excellent teacher
through dedication of time to
teaching and a highly effective
style, enabling significant skill
development throughout the
rotation.

Instructor Strengths
What are the strengths of this instructor? (These comments will be viewed by the instructor, but will
be anonymous and aggregated. For comments to be effective feedback, please be direct, specific, and
constructive. General comments such as “good instructor” are too non-specific to be of value.)
Instructor Improvements
How could this instructor improve? (These comments will be viewed by the instructor, but will be
anonymous and aggregated. For comments to be effective feedback please be direct, specific, and
constructive. General comments such as “bad instructor” are too non-specific to be of value.)
Confidential Comments, Educator
This area is for giving constructive or corrective feedback that you don't feel comfortable giving
directly. These comments are CONFIDENTIAL and will NOT go directly to the educator. They will
be forwarded ANONYMOUSLY to the program director(s). Please be thoughtful, professional, and
constructive in your feedback.
OPTIONAL CONFIDENTIAL COMMENT
If you are willing to be contacted by the clerkship director to address a particularly concerning issue,
please include your name and contact information below. This will only go the clerkship director
and/or the site director with the goal of appropriately addressing the raised concerns.
60

UCSF Academy of Medical Educators
TOP Observation Form­ Lecture
NAME: _____________________________________ OBSERVER: ___________________________
TOPIC: ______________________________________________________________________________
FOCUS OF OBSERVATION (discuss w/ faculty in advance):
INTRODUCTION
16. Introduced topic, stated objectives, offered
preview.
17. Gained attention and motivated learning.
18. Established climate for learning and for
participation.
61
OBSERVATIONS
BODY OF LECTURE OBSERVATIONS
19. Presented 3 – 5 main points in clear and
organized fashion.
20. Provided supporting materials, examples, and
summaries.
21. Content level
22. Effectively used visuals, handouts, and/or
demonstrations. Include AV problems (if any),
effective use of slides (set stage for each slide,
focused audience on important parts of slides),
use of pointer.
23. Varied presentations (Used blackboard, slides,
visuals).
24. Transitions between topics.
CONCLUSION OBSERVATIONS
25. Summarized major principles, key points
without introducing new materials.
26. Provided closure or stimulated further thought.

TEACHER DYNAMICS OBSERVATIONS
27. Exhibited enthusiasm and stimulated interest in
content.
28. Used appropriate voice, gestures, movement,
and eye contact. Avoidance of unconscious use
of repeated words (eg “um”, “ok”).
29. Encourage active participation.
30. Used questions to stimulate thought and
discussion. Response to questions (repeated or
rephrased question, concise answer).
DEBRIEF
4. ELICIT SELF-ASSESSMENT BY MENTEE FIRST.
5. SUMMARIZE YOUR ASSESSMENT OF MENTEE’S STRENGTHS AND YOUR RECOMMENDATIONS
(KEEP IN MIND AREAS OF FOCUS).
STRENGTHS
1.
2.
3.
6. ACTION PLAN (RESIDENT TO COMPLETE)
RECOMMENDATIONS
62

UCSF Academy of Medical Educators
TOP Observation Form­ Small Group
FACULTY NAME:_________________________________________DATE:_____________
GROUP SESSION:____________________________________________________________
Describe specific observations for each element of the discussion.
PROVIDED PREVIEW
NOTES
1. Introduced self and topic, offered
rationale for learning content and made
connection to larger course clear.
2. Stated objectives and provided preview
of session content and process.
3. Established positive learning climate and
expectations for participation.
4. Initiated discussion and captured
attention.
INVOLVED GROUP MEMBERS NOTES
5. Encouraged active and balanced
participation through in-class
assignments, sub-grouping or other
teaching techniques
6. Used questions and silences or posed
problems to stimulate thought and
discussion.
7. Exhibited enthusiasm and stimulated
interest in content
8. Managed group process issues
63

COVERED CONTENT NOTES
9. Progressed through content and
focused discussion on main points
10. Directed and paced discussion;
managed time for each section
11. Used teaching strategies to
stimulate thinking and clarify ideas
(e.g. provided analogies, examples or
supporting data; rephrased and
simplified complex statements;
modeled reasoning process
12. Used visuals to capture main ideas.
13. Summarized periodically and
bridged to next topic.
PROVIDED SUMMARY NOTES
14. Summarized key points (or asked
others) and provided closure.
15. Bridged to larger course or next
small group session.
16. Reviewed learning issues and made
assignments
17. Elicited feedback on session
STRENGTHS
64
RECOMMENDATIONS

GME Evaluation Task Force Recommendation
PROGRAM EVALUATION BY FACULTY AND RESIDENTS
The ACGME requires that residents and faculty evaluate the training program at least
annually and that the evaluation results be used at the Annual Program Review meeting to
assess the program’s performance and set future goals. Rather than recommend a specific
evaluation form to be used by all UCSF GME programs, the Evaluation Task Force offers
these recommendations to help programs design their own forms.
1. Align the program evaluation with the overall objectives of the program. For example,
if the program aims to ‘create the next generation of academic physicians in . . . “, it
would be important to address the career intention of the residents and the processes
the program uses to encourage or support academic careers.
2. Consider which program objectives and outcomes would be valid to assess using a
survey of residents and faculty. The development of values, career priorities and
intentions, perceived competency, and preparation for practice are examples of
outcomes that can be surveyed.
3. Consider which methods and processes your program uses to achieve its objectives
and survey faculty and residents on the perceived quality of these activities. Examples
include the perceived quality of didactic experiences, clinical rotations, training sites
and their resources; engagement of the faculty in general and at different training sites;
support for research, pathways and individualized programs of study; access to the
program director and other leaders; assessment of the learning environment.
4. Allow open‐ended responses to questions about the programs strengths, opportunities
for improvement, and specific suggestions for improvement.
5. Emphasize the confidential nature of the survey and that the results will be shared
only in composite form at the Annual Review Meeting.
65

GME Evaluation Task Force
POTENTIAL MEASURES USED IN GME PROGRAM EVALUATION
The Curriculum The Faculty Evidence of
Resident Learning,
ACGME
Competencies
Action plan from last
Annual Program
Review
Action plan from last
Annual Program
Review
Action plan from last
Annual Program
Review
ACGME survey items ACGME survey items Global assessments by
faculty
Confidential program Confidential program MK: In‐training exam
evaluation by faculty evaluation by faculty results
and residents
and residents
Clinical activity Confidential
Patient Care: Mini‐
evaluations
evaluations of faculty CEX, Focused
teaching
Assessments
Didactic and skill Faculty development Professionalism / ICS:
laboratory evaluations activities: number and
type
360 degree evaluations
In‐training exam Faculty mentorship PBLI
results including and advising
Critical appraisal
content area scores
assessments
Faculty scholarly
activity and
involvement of
residents
Program Resources,
Achievement of
Overall Objectives
Action plan from last
Annual Program
Review
ACGME survey items
Confidential program
evaluation by faculty
and residents
Duty hours, stress and
fatigue
Post‐residency career
trajectory
Quality of rotation
sites: clinical volume,
resources, support
Clinical teaching
assessments
SBP
Performance on board
M&M presentation, QI certifying examination
project, or other
activity
Patient or procedure Protected time for
log
faculty teaching
Scholarly project(s) Funding to support
PD, PC, faculty leaders
and program
administrators
Portfolio including Resident salaries and
reflections and benefits including
products
travel support,
protected time for
electives / Pathways
Individual Learning Resident recruitment
Plan
and retention
Alumni survey
66

Area / Evaluation
Instrument
GME Evaluation Task Force Recommendation
PROGRESS REPORT FOR ANNUAL PROGRAM REVIEW
The Curriculum (completed by residents)
Clinical activity evals
UCSF/MZ
SFGH
VAMC
Other
Didactic evaluations
Seminars
Skill labs
In‐training Exam
Total Score
Content Area A
Content Area B
Content Area C
Definition Mean (SD)
or Percent
3 5‐option items rated at end of
each rotation
4 5‐option items rated after each
session
Standardized score where 200 is
the mean and 20 is the SD based on
national norms for each PGY
67
4.04 (0.50)
3.96 (0.48)
4.10 (0.35)
3.60 (0.88)
3.88 (0.42)
4.50 (0.38)
205 (20)
180 (25)
208 (18)
202 (22)
Standard
(Goal)
4.0
4.0
4.0
4.0
4.0
4.0
200
200
200
200
Performance
Relative to
Standard
Met
Not Met
Met
Not Met
Not Met
Met
Met
Not Met
Met
Met
The Faculty (completed by residents unless otherwise noted)
Confidential evals
Average of 10 5‐option items
UCSF/MZ
4.04 (0.20) 4.0
Met
SFGH
4.08 (0.18) 4.0
Met
VAMC
4.10 (0.24) 4.0
Met
Volunteer
4.00 (0.33) 4.0
Met
ACGME survey items re:
faculty involvement
Average % Yes (range of 6 items) 92% (88 ‐ 96) 90% Met
Peer feedback sessions Percent of faculty who were
observed at least once during past
year
28% 50% Not Met
Evidence of Resident Learning, including ACGME Competencies
Global Evaluations
PGY 1
Average of 6 9‐option items
6.8 (1.1) 5
Met
PGY 2
completed after each rotation 7.1 (0.9) 5
Met
PGY 3
7.6 (0.8) 5
Met
PGY 4 …
8.3 (0.6) 5
Met
Medical Knowledge: MCQ Standardized score where 200 is
PGY 1, etc.
the mean and 20 is the SD based on
national norms for each PGY
220 (10) 200
Met
Patient Care: Mini‐CEX Average of 7 9‐option items
PGY 1, etc.
completed by faculty
7.7 (1.2) 5
Met
Patient Care: Focused
Assessment of Skills
PGY 1&2
PGY 3&4
Percent of checklist items
performed correctly
71.5%
83.2%
70%
90%
Met
Not Met

Prof / ICS: Team, Self
PGY 1, etc.
Average of 9 9‐option items
68
8.2 (0.7)
Prof / ICS: Patient Survey Average of 10 5‐option items
PGY 1, etc.
4.5 (0.2) 4.0
Met
PBLI – Student teaching Average of 6 5‐option items
PGY 1, etc.
4.2 (0.5) 4.0
Met
PBLI – Critical appraisal Percent of checklist items met
PGY 1, etc.
90% 80% Met
SBP – M&M evaluation Average of 8 5‐option items
PGY 1, etc.
3.8 (1.2) 4.0
Met
Case logs – Procedure A % residents in PGY year meeting
PGY 1, etc.
targets based on national
100% 100% Met
Case logs – Procedure B percentiles
PGY 1, etc.
Case logs – Procedure C
100% 100% Met
PGY 1, etc.
Scholarly Project
88% 100% Met
Oral Presentation Score Total points out of 20
18.2
16
Met
Meeting Presentations % residents completing by PGY4 100% 90% Met
Manuscripts Submitted % residents completing by PGY4 67% 80% Not Met
Favorable Review
% favorable (if submitted)
33% 80% Not Met
Program Resources, Achievement of Overall Objectives
Duty hours – ACGME survey
items
Average % residents compliant 90% 90% Not Met
Duty hours – GME reporting Average % residents compliant 86% 90% Not Met
ACGME survey items re:
program
Average % Yes 92% 90% Met
Program Eval by Residents Average of 8 5‐option answers 4.3 4.0 Met
Program Eval by Faculty Average of 8 5‐option answers 4.0 4.0 Met
Program Eval by Alumni Average of 8 5‐option answers 4.6 4.0 Met
Written Board Exam Pass
Rate on First Attempt
Percent grads passing last year 100% 100% Met
Oral Board Exam Pass Rate
on First Attempt
Career Trajectory
Percent grads passing last year 88% 100% Not Met
Fellowship Applicants % Applying to Fellowships
67% 50% Met
Fellowship Acceptance % Matched of Those Applying 100% 100% Met
Case Volume: Procedure A Case volume available for resident
(targeted at last review) training based on OR logs
Total Available
240 250 Not Met
UCSF/MZ
80
100 Not Met
SFGH
60
50
Met
VAMC
46
50 Not Met
Extramural
54
50
Met
Adapted from Knight DA, Vannatta PM, O’Sullivan PS. A Process to Meet the Challenge of
Program Evaluation and Program Improvement. ACGME Bulletin 2006 (Sept): 5‐8.
5
Met

APPENDIX A
ACGME Common Program Requirements – Section IVB
ACGME Competencies
The program must integrate the following ACGME competencies into the curriculum:
• Patient Care
Residents must be able to provide patient care that is compassionate, appropriate, and
effective for the treatment of health problems and the promotion of health. Residents
are expected to:
[as further specified by the RRC]
• Medical Knowledge
Residents must demonstrate knowledge of established and evolving biomedical, clinical,
epidemiological and social‐behavioral sciences, as well as the application of this
knowledge to patient care. Residents are expected to:
[as further specified by the RRC]
• Practice‐based Learning and Improvement
Residents must demonstrate the ability to investigate and evaluate their care of patients,
to appraise and assimilate scientific evidence, and to continuously improve patient care
based on constant self‐evaluation and life‐long learning. Residents are expected to
develop skills and habits to be able to meet the following goals:
o identify strengths, deficiencies, and limits in one’s knowledge and expertise;
o set learning and improvement goals;
o identify and perform appropriate learning activities;
o systematically analyze practice using quality improvement methods, and
implement changes with the goal of practice improvement;
o incorporate formative evaluation feedback into daily practice;
o locate, appraise, and assimilate evidence from scientific studies related to their
patients’ health problems;
o use information technology to optimize learning; and,
o participate in the education of patients, families, students, residents and other
health professionals.
• Interpersonal and Communication Skills
Residents must demonstrate interpersonal and communication skills that result in the
effective exchange of information and collaboration with patients, their families, and
health professionals. Residents are expected to:
o communicate effectively with patients, families, and the public, as appropriate,
across a broad range of socioeconomic and cultural backgrounds;
69

o communicate effectively with physicians, other health professionals, and health
related agencies;
o work effectively as a member or leader of a health care team or other professional
group;
o act in a consultative role to other physicians and health professionals; and,
o maintain comprehensive, timely, and legible medical records, if applicable.
• Professionalism
Residents must demonstrate a commitment to carrying out professional responsibilities
and an adherence to ethical principles. Residents are expected to demonstrate:
o compassion, integrity, and respect for others;
o responsiveness to patient needs that supersedes self‐interest;
o respect for patient privacy and autonomy;
o accountability to patients, society and the profession; and,
o sensitivity and responsiveness to a diverse patient population, including but not
limited to diversity in gender, age, culture, race, religion, disabilities, and
sexual orientation.
• Systems‐based Practice
Residents must demonstrate an awareness of and responsiveness to the larger context
and system of health care, as well as the ability to call effectively on other resources in
the system to provide optimal health care. Residents are expected to:
o work effectively in various health care delivery settings and systems relevant to
their clinical specialty;
o coordinate patient care within the health care system relevant to their clinical
specialty;
o incorporate considerations of cost awareness and risk‐benefit analysis in patient
and/or population‐based care as appropriate;
o advocate for quality patient care and optimal patient care systems;
o work in interprofessional teams to enhance patient safety and improve patient
care quality; and
o participate in identifying system errors and implementing potential systems
solutions.
70

Resident Evaluation
1. Formative Evaluation
APPENDIX B
ACGME Common Program Requirements – Section V
The faculty must evaluate resident performance in a timely manner during each rotation or
similar educational assignment, and document this evaluation at completion of the
assignment.
The program must:
(1) provide objective assessments of competence in patient care, medical knowledge,
practice‐based learning and improvement, interpersonal and communication skills,
professionalism, and systems‐based practice;
(2) use multiple evaluators (e.g., faculty, peers, patients, self, and other professional staff);
(3) document progressive resident performance improvement appropriate to educational
level; and,
(4) provide each resident with documented semiannual evaluation of performance with
feedback.
The evaluations of resident performance must be accessible for review by the resident, in
accordance with institutional policy.
2. Summative Evaluation
The program director must provide a summative evaluation for each resident upon
completion of the program. This evaluation must become part of the resident’s permanent
record maintained by the institution, and must be accessible for review by the resident in
accordance with institutional policy. This evaluation must document the resident’s
performance during the final period of education, and verify that the resident has
demonstrated sufficient competence to enter practice without direct supervision.
______________________________________
71

Faculty Evaluation
1. At least annually, the program must evaluate faculty performance as it relates to the
educational program.
2. These evaluations should include a review of the faculty’s clinical teaching abilities,
commitment to the educational program, clinical knowledge, professionalism, and scholarly
activities.
3. This evaluation must include at least annual written confidential evaluations by the
residents.
______________________________________
Program Evaluation and Improvement
1. The program must document formal, systematic evaluation of the curriculum at least
annually. The program must monitor and track each of the following areas: resident
performance; faculty development; graduate performance, including performance of
program graduates on the certification examination; and, program quality. Specifically:
(1) Residents and faculty must have the opportunity to evaluate the program confidentially
and in writing at least annually, and
(2) The program must use the results of residents’ assessments of the program together with
other program evaluation results to improve the program.
2. If deficiencies are found, the program should prepare a written plan of action to
document initiatives to improve performance. The action plan should be reviewed and
approved by the teaching faculty and documented in meeting minutes.
72

Surgical Competency
Ob/Gyn - UCSF
APPENDIX C
EXAMPLES OF FOCUSED ASSESSMENT TOOLS
Evaluator: Date:
Resident: PGY: 1 2 3 4
Rotation:
Diagnosis:
UCSF Ob/Gyn Surgical Skill Checklist
73
Surgical Skills Assessment
0 = poorly or never
3 = majority or well
1 = sometimes or marginal
4 = always or excellent
2 = usually or average
Procedure: (R3) Total Abdominal Hysterectomy
Knew patent history / surgical indication
Operative Checklist
Necessary lines in place (intravenous, foley)
Rating X = not seen or indicated 1 = performed but poorly
Patient positioned correctly on table
Key:
0 = not performed but indicated 2 = performed correctly
Proper stirrups/retractor used for exposure
Lights positioned
1. Discuss the indications for hysterectomy X 0 1 2
Observed sterile technique
2. Discuss the indications for oophorectomy in conjunction with
abdominal hysterectomy
X 0 1 2
Knew names of instruments
3. Discuss the post-operative management of a patient status
post TAH
X 0 1 2
Knowledge of anatomy
4. Choice of abdominal incision X 0 1 2
Instrument handling
5. Ligation of round ligament X 0 1 2
Respected tissue
6. Anterior & posterior leaf of broad ligament opened
X 0 1 2
Moves not wasted
7. Creation of broad ligament window X 0 1 2
Kept flow of operation / thought ahead
8. Identification of the ureter X 0 1 2
Used assistants well
9. Ligation of uteroovarian ligament vs infundibulopelvic ligament
(+/-BSO)
X 0 1 2
Worked well with personnel
10. Double ligation of pedicles X 0 1 2
Worked well as primary surgeon
11. Sharp dissection of bladder flap X 0 1 2
12. Skeletonize uterine vessels X 0 1 2
TOTAL =
13. Cardinal ligament ligation X 0 1 2
14. Uterosacral ligament ligation X 0 1 2 STRENGTHS:
15. Vaginal cuff closure X 0 1 2
16. Vaginal cuff suspension X 0 1 2
17. Evaluation for hemostasis X 0 1 2
AREAS FOR IMPROVEMENT:
Modified from:
For administrative use
AJOS 1997; 173:226-230 Attending Signature:
AJOS 1994; 167:423-427
Entered by: _________
Date: ______________
Resident Signature:
Rating
Key:

EMERGENCY MEDICINE RESIDENCY PROGRAM
RESUSCITATION COMPETENCY FORM
Resident: ____________________________ Date:___________________________
Attending Physician: ____________________________ Location: ________________________
According to the ACGME, a major resuscitation is patient care for which prolonged physician attention is needed and interventions such
as defibrillation, cardiac pacing, treatment of shock, intravenous use of drugs (e.g., thrombolytics, vasopressors, neuromuscular
blocking agents), or invasive procedures (e.g., cut downs, central line insertion, tube thoracostomy, endotracheal intubations) that are
necessary for stabilization and treatment.
I. CLINICAL (Patient Care/Medical Knowledge)
Primary Survey: Y N NA
Airway assessed initially Y N NA
Breathing then assessed
Y N NA
Oxygen started for respiratory distress
Circulation assessed Y N NA
Initial interventions Y N NA
Protocol or treatment guideline
followed
Y N NA
Patient reassessed frequently Y N NA
Secondary Survey (head to toe
exam):
Y N NA
Procedures performed competently Y N NA
74
Comments:
II. ORGANIZATION (Communication/Professionalism/Systems-Based Practice)
Comments:
Assigned roles Y N NA
Communicates effectively Y N NA
Asked for help when needed Y N NA
Maintains situational awareness Y N NA
Appropriate handoff (SBAR) Y N NA
□ COMPETENT □ NEEDS IMPROVEMENT

EMERGENCY MEDICINE RESIDENCY PROGRAM
AIRWAY MANAGEMENT COMPETENCY FORM
Resident: ____________________________ Date:___________________________
Attending Physician: ____________________________ Location: ________________________
I. PREPARATION:
Personally assembled and tested all necessary equipment (e.g.,
blades, ET tubes, oral/nasal airways, suction, BVM, etc.)
Properly positioned himself/herself at the head of the bed and all
necessary equipment within arm’s reach
Verbalized an appropriate “Plan B” should initial attempts at airway
management fail (e.g., use of a different type blade, gum elastic
bougie, cric., etc.)
II. MEDICATION MANAGEMENT:
Ordered an appropriate induction and paralytic drug,
demonstrating understanding of the particular
indications/contraindications for this drug
Ordered appropriate post-intubation sedation medication,
demonstrating understanding of the particular
indications/contraindications for this drug
III. AIRWAY TECHNIQUE:
75
Y N
Y N
Y N NA
Y N NA
Y N NA
Properly positioned patient/head Y N NA
Effectively performed bag-mask-valve ventilation
Maintained a patent airway (with good positioning, oral/nasal
trumpets, etc.) prior to intubation
Y N NA
Y N NA
Properly applied cricoid pressure Y N
Demonstrated proper use of a laryngoscope and proper ET tube
placement
Y N
Confirmed proper tube placement with: - Auscultation Y N NA
- End-tidal CO2 Y N NA
- CXR
Y N NA
Applied necessary alternate rescue airway technique(s) Y N NA
IV. VENTILATOR MANAGEMENT:
Ordered appropriate initial ventilator settings Y N NA
V. DOCUMENTATION:
Medications ordered on order sheet Y N
Procedure documented in chart Y N
□ COMPETENT □ NEEDS IMPROVEMENT
Comments:
Comments:
Comments:
Comments:
Comments:

Structured Clinical Observation: Resident Interview
LPH&C Medication Management Clinic
John Q. Young, MD, MPP, UCSF Department of Psychiatry, v. 2.5.08
May be used or adapted outside UCSF only with permission of the author jqyoung@lppi.ucsf.edu
Resident Name:_________________________ Attending Name:___________________________
Date:
Instructions: 1. Each resident observed Q4 weeks. 2. Attending checks one box for each row and writes comments at
bottom. 3. Attending reviews with resident and then places in John Young’s box in LP‐281 who will give copy to
resident.
Pharmacotherapy Task
Reviews chart
Greets patient with respect & warmth
Begins on time
Maintains frame
Establishes rapport
Initial open ended question
Obtains interval history with focus on target
symptoms, medical or medication changes,
intercurrent psychosocial stressors, progress in
therapy.
Assesses treatment response
Encourages ventilation of feelings related to illness.
Inquires about other treatments/treaters
Assesses substance use/abuse
Assesses adherence, including number of doses
missed in past week and barriers.
Monitors for adverse effects (Sg/Sx, Labs, AIMS, Wt.,
BP), specifically for those associated with prescribed
medications.
MSE appropriately focused
Assesses risk for violence to self and others
If response less than expected, systematic approach to
DDx
Updates treatment plan based on diagnosis, phase of
illness, efficacy and response, adverse effects, & risk
assessment
Modifies treatment plan for less than expected
responders
Develops plan to address adherence if needed
Develops plan to manage adverse effects, if applicable
0
NA
76
1
Not
Done
2
Done with
suggestions
for
improvement
3
Done well
(meets
expectatio
ns)
4
Done
extraordinarily
well – inspires
me to do the
same!

Educates patient about diagnosis, prognosis,
treatment, and/or adverse effects
Provides patient with simple advice on what can do
to help self (e.g., exercise, sleep hygiene).
Solicits and addresses patient’s questions
Conveys hope and optimism and provides
reassurance
Appropriate follow up, incl. labs/tests, consults, next visit
Documentation sufficient
Informs other tx team members of plan, esp. therapists.
77

Structured Clinical Observation: Resident Interview
LPH&C Medication Management Clinic
John Q. Young, MD, MPP, UCSF Department of Psychiatry, v. 8.1.07
May be used or adapted outside UCSF only with permission of the author jqyoung@lppi.usf.edu
Key feedback points, including what done well and at least one task to work on:
DDx for less than Expected
Response
Modify treatment plan for less
than Expected Response
• Incorrect primary diagnosis?
• Correct primary diagnosis, but insufficient treatment?
• Poor adherence?
• Under‐ or un‐treated comorbidity (e.g., substance abuse, axis I, axis II
etc…)?
• Intervening stressor
• Adverse effects of treatment?
• Alliance ruptured?
A. Pharmacologic Interventions
• Address adherence
• Reassess dose and duration
• Consider a switch to an alternative treatment
• Augment with evidence based second and third line
pharmacologic treatments
• Treat comorbidities
B. Nonpharmacologic Interventions
• Provide further education
• Provide opportunity to “ventilate” with active listening
• Provide reassurance
• Provide specific psychotherapy
• Refer for psychotherapy
• Behavioral intervention (e.g., sleep hygiene)
• Improve alliance
• Improve treatment of comorbidities such as substance abuse
• Involve family members
78

Communication competency-
UCD OBG
UC Davis Ob/Gyn Informed Consent Checklist
Evaluator: Date:
Resident: PGY: 1 2 3 4
Rotation:
Diagnosis:
Procedure: INFORMED CONSENT
Rating X = not seen or indicated 1 = performed but poorly
Key: 0 = not performed but indicated 2 = performed correctly
1.Know proper indications for procedure X 0 1 2
2.Know alternatives
3. Establishes rapport with patient
X 0 1 2
X 0 1 2
4. Properly describes procedure in understandable terms X 0 1 2
5. Realistically explains risks of procedure
6.Discusses benefits of procedure
7. Discusses alternatives to procedure
8. Checks for patient understanding often
9. Explains preop procedure
Communication check list
X 0 1 2
X 0 1 2
X 0 1 2
X 0 1 2
X 0 1 2
79
Communication Skills List
0 = poorly or never
3 = majority or well
Rating Key:
1 = sometimes or marginal
2 = usually or average
4 = always or excellent
10.Explains hospital procedure
11. Explains follow up
X 0 1 2
X 0 1 2
12. Assess patient questions X 0 1 2 TOTAL =
X 0 1 2
X 0 1 2
X 0 1 2
STRENGTHS:
AREAS FOR IMPROVEMENT:
Attending Signature:
Resident Signature:
Communicates clearly
Listens willingly and attentively
Answers questions and provides explanations
Respects patient does not demean
Uses respectful language
Compassion and kind to patient and family
Attentive to details of patient comfort
Worked well with personnel
Nonverbal: shows interest

Long Form
UNIVERSITY OF NORTH CAROLINA
GRIEVING COMPETENCY INSTRUMENT
Directions: Please indicate whether the physician completed the stated actions, with
Y = completed (Yes) or N = did not complete (No)
The Physician…
G—Gather
1. Ensured that all important survivors were present prior to delivery of the death notification.
R—Resources
2. Inquired about supportive resources.
3. Facilitated access to supportive resources.
I—Identify
4. Clearly stated the name of the patient.
5. Clearly introduced herself/himself.
6. Clearly stated his/her role in the care of the patient.
Check for Understanding
7. Determined the level of knowledge the survivors possessed prior to their arrival in the
waiting room.
8. Provided an appropriate opening statement (i.e., avoided bluntly stating death of patient).
9. Used preparatory phrases to forecast the news of death.
E—Educate
10. Clearly indicated the chronology of events leading up to the death of the patient.
11. Clearly indicated the cause of death in an understandable manner.
12. Used language appropriate for the survivor’s culture and educational level.
13. Provided a summary of important information to ensure understanding.
V—Verify
14. Used the phrase “dead” or “died.”
15. Avoided using euphemisms.
16. Avoided medical terminology/jargon or clearly explained such terms when used.
Space
17. Was attentive and not rushed in his/her interaction with survivor.
18. Paused to allow the family to assimilate the information before discussing details.
80

I—Inquire
19. Allowed the survivor to react to the information and ask questions or express concerns.
20. Encouraged the survivor to summarize important information to check for understanding.
21. Immediately but appropriately corrected any misconceptions of the survivor.
N—“ Nuts and bolts”
Explained and addressed the following details of the patientʹs post‐mortem care adequately.
22a. Organ donation
22b. Need for an autopsy
22c. Funeral arrangements
22d. Personal effects
G—Give
25. Established personal availability to answer questions for the survivor at a later date.
26. Provided the survivor appropriate information to contact the physician at a later time.
27. Provided the survivor appropriate information to contact resuscitation or post‐mortem
care providers.
81