Faculty Members' Views on the Usefulness of Various Assessments

<strong>Faculty</strong> Members’
<strong>Views</strong> on the
Usefulness of
Various
Assessments
Jeremy Penn,
Ph.D.
jpenn3@unl.edu

<strong>Faculty</strong> members
“I have sometimes thought of [the
University] as a series of individual
Office of Undergraduate Studies
faculty entrepreneurs held together by a
common grievance over parking” (Kerr,
1964).

Call to action
Office of Undergraduate Studies
“…every American will need to get more than a
high school diploma…That is why we will
provide the support necessary for you to
complete college and meet a new goal: by
2020, America will once again have the
highest proportion of college graduates in the
world.” (President Obama, Address to Joint
Session of Congress, February 24, 2009)

Improvement efforts
• International
• Federal Government
• State Governments
• Testing companies
• Consortia of institutions
• Other agencies
• Institutions themselves
Office of Undergraduate Studies

<strong>Faculty</strong> involvement
Office of Undergraduate Studies
• Tends to be limited (Banta, 2002; Blackburn &
Lawrence, 1995; Bok, 2006; Watty, 2003; Wergin &
Swingen, 2000)
However:
• <strong>Faculty</strong> members are committed to quality
(Anderson, 2006)
• <strong>Faculty</strong> members give more effort to institutional
service than they believe administrators wanted
(Blackburn & Lawrence, 1995)

Purpose
Office of Undergraduate Studies
• Learn more about faculty members’ views on
assessment instruments
• Find ways to get faculty members more involved in
improvement efforts
– Ensure the success of those efforts

Explanatory mixed methods case study
research design
Conceptualization
Stage
Quantitative
Methods Stage
Quantitative
Data Analysis
Quantitative
Inferences
Meta-
Inferences
Conceptualization
Stage
Qualitative
Methods Stage
Qualitative
Data Analysis
Qualitative
Inferences
Office of Undergraduate Studies
Teddlie & Tashakkori, 2006

Quantitative methods
• Survey based on prior research
• Pilot study
• Stratified random sample from a single large
research university (~1400 faculty members)
– <strong>Faculty</strong> members who had not taught an
undergraduate course in last 2 years were
removed
• Effective response rate = 39% (216 completed
surveys)
Office of Undergraduate Studies

Qualitative methods
• Multicase study of 5 teaching award winners
• Two tape recorded hour-long interviews
• Multiple observations of teaching between
interviews
Office of Undergraduate Studies

Outline
• Data on assessment measures
– Quantitative and Qualitative findings
• Data on faculty members
– Quantitative and Qualitative findings
• Summary and discussion
Office of Undergraduate Studies

Assessment measures
Alumni surveys
Employment rates
Enrollment numbers
Graduation rates
Informal conversations with other faculty members
Informal conversations with students
National measures (e.g., NSSE, CLA, GRE, etc.)
Reviews by external experts
Standardized tests or licensure exams
Students’ course evaluations
Summaries of locally developed assessments
U.S. News and World Report rankings
Office of Undergraduate Studies

Results: Mean rating of usefulness of
measures of teaching and learning
Weighted n = 1062 M SD SE
Informal conversations with students 5.53 .63 .02
Informal conversations with other faculty members 5.20 .86 .03
Students’ course evaluations 4.75 1.09 .03
Employment rates 4.27 1.70 .05
Graduation rates 4.13 1.65 .05
Enrollment numbers 3.62 1.45 .04
Alumni surveys 3.40 1.89 .06
Reviews by external experts 3.37 1.89 .06
National measures (e.g., NSSE, CLA, GRE, etc.) 3.05 1.81 .06
Summaries of locally developed assessments 2.99 1.79 .06
Standardized tests or licensure exams 2.78 1.81 .06
U.S. News and World Report rankings 2.69 1.41 .04
Office of Undergraduate Studies
1 = Have not used; 2 = very useless; 3 = somewhat useless; 4 = neutral; 5 = somewhat useful; 6 = Very
useful

Results: Usage of assessment measures
Office of Undergraduate Studies
Weighted n = 1062 Percent “Have NOT used”
Informal conversations with students 1%
Informal conversations with other faculty members 1%
Students’ course evaluations 2%
Employment rates 18%
Graduation rates 17%
Enrollment numbers 14%
Alumni surveys 34%
Reviews by external experts 33%
National measures (e.g., NSSE, CLA, GRE, etc.) 36%
Summaries of locally developed assessments 38%
Standardized tests or licensure exams 44%
U.S. News and World Report rankings 26%

Two paradigms (adapted from Ewell,
2008)
Assessment for
Improvement
Office of Undergraduate Studies
Assessment for
Accountability
Purpose Formative Summative
Stance Internal External
Predominant ethos Engagement Compliance
Instrumentation Multiple Standardized
Nature of evidence Quan or Qual Quan only
Reference points Over time, goals
Communication of
results
Multiple internal
and media
Comparative or
fixed
Public
communication
Use of results Feedback loops Reporting

Two paradigms as a continuum
Office of Undergraduate Studies

Usefulness and accountability
Office of Undergraduate Studies

Assessment qualities
Top 2 measures
on usefulness
Bottom 2
measures on
usefulness
Speed of
feedback
/ results
Proximity
to practice
Fast Near
Slow Far
Selection of
items /
questions
Clear and
Involved
Unknown
and
disconnected
Office of Undergraduate Studies
Actionability
of
results
Obvious
Mysterious

Qualitative results
Office of Undergraduate Studies
• “Institutional quality” was too far removed from the
everyday experience of a faculty member
– Could, however, conceptualize department
quality
• “Change every time I teach it”
• High involvement in classroom assessment
– Wide range of measures and processes used
– Recognized the higher workload but felt it was
worth it

Discussion
Office of Undergraduate Studies
• <strong>Faculty</strong> members and assessment leaders need to be
willing to make some compromises
• Small changes to existing measures can make a big
difference
• Try to avoid going too “global”
• Enhance communication between faculty members
and assessment leaders
• The perfect instrument may not exist
– “Convergence of evidence” (Kathleen Yancey)

Convergence of evidence (Kathleen
Yancey)
Office of Undergraduate Studies
• Inputs (e.g., syllabus, student entry assessment)
• Perceptions (e.g., course evaluations, surveys)
• Behaviors (e.g., NSSE)
• Outputs (e.g., standardized measures, student work)

Examination of faculty members
Office of Undergraduate Studies
• First set of results examined the measures
– Will now examine faculty members
• Cluster analysis
– Group together faculty members with similar
response patterns

Results: Mean scores by cluster
Office of Undergraduate Studies
n = 216 Highly Not Actively
engaged engaged disengaged
Informal conversations with students 5.52 5.56 5.40
Informal conversations with other faculty members 5.29 5.16 5.28
Students’ course evaluations* 4.82 4.78 4.26
Employment rates*** 5.04 4.86 1.30
Graduation rates*** 4.92 4.52 1.35
Enrollment numbers*** 4.45 3.64 2.05
Alumni surveys*** 4.71 2.83 1.19
Reviews by external experts*** 4.51 2.64 2.35
National measures (e.g., NSSE, CLA, GRE, etc.)*** 4.59 2.05 1.35
Summaries of locally developed assessments*** 4.24 2.52 1.26
Standardized tests or licensure exams*** 4.43 1.90 1.26
U.S. News and World Report rankings*** 3.76 2.21 1.77

Engagement levels
Office of Undergraduate Studies
• Highly engaged: passionate, innovative, connected
to institution
• Not engaged: sleepwalking through the day, checked
out, “zombies”
• Actively disengaged: actively act out their
unhappiness, undermine others, they don’t know
what is expected of them
(Source: Crabtree, 2004)

Cluster details
Office of Undergraduate Studies
• Clusters were not significantly different on tenure
status or gender
• Clusters were not significantly different on “effort
to improve courses”
• Highly engaged gave significantly more effort to
improve teaching and learning at department and
institutional level than not engaged or actively
disengaged (Hedge’s G from .38 to .51)

Within cluster paired t-test on effort
Office of Undergraduate Studies
Highly
engaged
Not
engaged
Actively
disengaged
-Department -Institution
Course 0.88*** 1.60***
Department 0.73***
Course 1.31*** 2.08***
Department 0.77***
Course 1.35*** 2.14***
Department 0.79***

Other differences by cluster
Office of Undergraduate Studies
• Highly engaged (compared to Actively disengaged)
– Agreed more with “quality as excellence”
– Higher preference for improvement activities
– Taught fewer (but > 0) undergraduate courses
• Highly engaged (compared to Not engaged)
– Agreed more strongly improvement activities
were rewarded fairly
– Agreed more strongly on access to resources for
improvement activities

Qualitative results
Office of Undergraduate Studies
• High involvement in classroom assessment
– As teaching tool
– As feedback mechanism
– “less worried about grades and all that and more worried
that they really understand the material”
– Learning to critique others’ work is important skill in
many disciplines
• Tenure
– “freedom to push the envelope”
– “protect” tenure-track faculty
– Tenured have “more committee work and more
expectations along those lines”

Qualitative results
Office of Undergraduate Studies
• Workload
– Some days…“I can hardly stand up”
– Overlap between teaching, research & service
• Support
– Generally felt supported
– “program never has enough resources, in terms
of faculty and classrooms”
• Wanted to do a good job
– Committed to student success
– “Do well by the students”

Implications
Office of Undergraduate Studies
• Working with the highly engaged
– Almost anything works
– Enlist their help in engaging other two clusters
• Working with the not engaged
– Ensure assessment measures are as useful as
possible
– Rewards and resources may be beneficial
• Working with the actively disengaged
– Smallest group
– May be overwhelmed by other responsibilities
– Consider carefully what “involvement” means

Summary
Office of Undergraduate Studies
• Work together with faculty members
– Measures assessment leaders may find useful may
not be as useful to faculty members
– Avoid “me versus them”
• Perceived usefulness of assessment measures are
important
– Be willing to make changes to help tools be more
useful

Summary
Office of Undergraduate Studies
• Consider multiple strategies for working with faculty
members
– Selecting “useful” measures can help
– Consider carefully how faculty members are to be
engaged in assessment
– Encourage assessment to be included as part of
appointments
• Build on interest in improving courses

Discussion
Office of Undergraduate Studies
• What practices have you found helped faculty
members find assessment measures more useful?
• How is “engagement in assessment” defined at your
institution?
• What strategies have been helpful in engaging
faculty members in assessment at your institution?

References
Office of Undergraduate Studies
Anderson, G. (2006). Assuring quality / resisting quality assurance: Academics responses’ to
‘quality’ in some Australian universities. Quality in Higher Education, 12(2), 161 – 173.
Banta, T. W. (2002). A call for transformation. In T. W. Banta and Associates (Eds.), Building a
scholarship of assessment (pp. 284 – 292). San Francisco: Jossey-Bass.
Blackburn, R. T., & Lawrence, J. H. (1995). <strong>Faculty</strong> at work: Motivation, expectation, satisfaction.
Baltimore: Johns Hopkins University Press.
Bok, D. (2006). Our underachieving colleges: A candid look at how much students learn and why they should
be learning more. Princeton: Princeton University Press.
Crabtree, S. (June 11, 2004). Getting personal in the workplace. Gallup Management Journal.
Ewell, P. T. (2008). Assessment and accountability in America today: Background in context. In
V. M. H. Borden & G. R. Pike (Eds.). Assessing and accounting for student learning: Beyond the
Spellings Commission. New Directions for Institutional Research. San Francisco: Jossey-Bass.
Kerr, C. (1964). The uses of the university. In W. S. Smith & T. Bender (2008), American higher
education transformed, 1940-2005: Documenting the National Discourse. Baltimore: Johns Hopkins,
48 – 51.
Teddlie, C. & Tashakkori, A. (2006). A general typology of research designs featuring mixed
methods. Research in the Schools, 13(1), 12 – 28.
Watty, K. (2003). When will academics learn about quality? Quality in Higher Education, 9(3), 213 –
221.
Wergin, J. F., & Swingen, J. N. (2000). Departmental assessment: How some campuses are effectively
evaluating the collective work of faculty. Washington, DC: American Association for Higher
Education.
Yancey, K. (2009). E-portfolios and assessment. Presentation at the University of Nebraska-Lincoln.