Formulating Qualitative Assessment Questionnaires Appropriate to

Alignment and AccountabilityFormulating QualitativeAssessment QuestionnairesPresenter: Lilli EngleAmerican University Center of Provence, FranceFacilitators:Micheal Vande Berg, CIEE & Patricia Martin, SwarthmoreCollegeCIEE ConferencePhiladelphia, Nov. 2010

Reconciliation of OppositesConsumer contentmentorEducational challenge

Reconciliation of OppositesSatisfaction-based questionnaires oftensimply refer students back to their ownpre-departure expectations

Reconciliation of OppositesStudents are asked to examine the qualityof their term abroad through the lens ofaccepted American standards**of teaching style, internet access, staff availability, housing comfort

Reconciliation of OppositesDifficult choiceCater to familiar client-centered expectations?or …

Reconciliation of OppositesStretch the limitsof what a student can appreciate in the name ofan educational experience by its very natureand purpose different from that at home

Reconciliation of OppositesStudent interest and satisfactionare essential to program survival

Reconciliation of OppositesToday’s challengeBring more appropriate criteria tothe consideration of “satisfaction”

Reconciliation of OppositesCriteria more in keeping withthe ideals of intl. education**Founded on the rigors, risks, and rewards of authentic intercultural experience

Assessing “Quality”in a Cross-Cultural ContextOutside the ethnocentric sphere ofaccepted cultural norms, whatvalue or meaning do the followingqualitative terms carry?

Assessing “Quality”in a Cross-Cultural ContextThe instructors were effectiveThe program was intellectually challengingI am satisfied with my study abroad experience

Assessing “Quality”in a Cross-Cultural ContextTo avoid the pitfall of ethnocentricjudgment, questioning may takeanother form…

Assessing “Quality”in a Cross-Cultural ContextAsking for more specific indicationsof how the students lived andbenefitted from the experience theprogram intentionally structured forthem

Aligning Actions and IdealsClarityof mission and purposeis key

Aligning Actions and Ideals1IntentionMission Statement4AssessmentEvaluationRecorded Outcomes2PerformanceProgram Implementation& Design3ResultsStudent Learning& Development

Aligning Actions and IdealsThe choice and formulation of questionsset the criteria for qualitative judgmentbased on the educational goals ofthe individual program

Engaging the ProcessConfirm the program’s mission statementConsider and list learning goalsDefine program componentsArticulate what each component intends to achieveEvaluate cross-cultural challenges per componentEstablish criteria of student engagement

Part I: Defining Mission, Goalsand Program DesignDefine:goalscomponentschallengesand

Part II: Aligning Components,Goals, Challenges, and QuestionsTargeting goals and outcomesBy implementing this component, whatdid the program hope to achieve?

Part II: Aligning Components,Goals, Challenges, and QuestionsAction VerbsTo encourage…To provide…To develop…To establish a link…

Part II: Aligning Components,Goals, Challenges and QuestionsEcho the objectivesin question formRemember, questions serve to assess to what extentthe program succeeded in realizing its goals

Part II: Aligning Components,Goals, Challenges and QuestionsQuestion:To what extent did the academicprogram establish a link between thecourse content and the study abroadlocation?

Part II: Aligning Components,Goals, Challenges and QuestionsIdentifyAdaptational challengesWeighing in the balance specific program contentand projected level of student preparedness

Part II: Aligning Components,Goals, Challenges and QuestionsEcho the challengesin question formChallenge: Adapting to a different teaching styleQuestion: To what extent were you able to adapt tothe more lecture-based teaching style of your hostnational professors?

Part III:The Ideal Student ComponentTargeting abstract student goalsWhat would the goals of anideal student be?

Part III:The Ideal Student ComponentList the general abstractstudent goals• Learn the host language• Cultivate friendships…

Part III:The Ideal Student ComponentTargeting behaviorWhat would the ideal studentwith this goal actually do?

Part III:The Ideal Student ComponentAction Verbs• Engage host nationals inconversation whenever possible• Read local press, watch local TV

Part III:The Ideal Student ComponentEcho student ideals inquestion form:In order to maximize my learning of the hostlanguage, I (check all that apply)• Engaged host nationals in conversation• Read local press• Watched local TV

RecapThe ideal questionnaire has at its corea clear vision of what the program is trying to achieve.Its questions reflect:• Intention: The goals of an ideal program• Performance: What that ideal program should do or provide• Results: What are the ideal outcomes• Evaluation/assessment : What does the ideal student* gain from theexperience and how?*Criteria for the ideal student : motivation, attitude, investment, effort, performance, growth

RecapEvaluating the questions themselves:• In what way is the question/answer meaningful to the program or to thehome university?• Does the question elicit the judgment of an empowered-consumer?• Are program goals and desired outcomes inherent in the question?• Is the responsibility of the student implicated in the evaluation of theexperience?• Are the standards of evaluation cross-culturally sensitive or weighted byUS standards or expectations?• Does the question invite criticism or negativity or simply leave room forstudent comments?

Questions: Overall ProgramExample I• How do you rate the program ?• Any suggestions for improvement ?• Any elements of the program that you would change?Example II– Consumer-based questions– Ethnocentric questions• Program staff members were accessibleECQCBQ• Each portion of the program built upon previous portions• Communication about program changes, activities, etc. was timelyand clear.

Questions: Overall ProgramExample III– Goal-based questionsEvaluate the amount of growth you experiencedin the following areas:GBQ• Understanding of different cultures• Different perspective on American culture• Greater understanding of the US in world affairs and history• Different perspective on your ethnic heritage• Understanding of your faith• Self-awareness• Maturity and self-confidence• Interpersonal skills and the ability to adapt• Intellectual development in the liberal artsSliding scale from 1-5 (from Great Growth to No Growth and Not Applicable)

Questions: AcademicsExample I – Consumer-based questionsRate the professor*• Quality of instruction• Attitude towards studentsCBQ• Accessible to students?Example II – Ethnocentric questionsPlease rate each course according to the following criteria:• Quality of the course*• Appropriateness of exams/assignments*• Level of difficulty as compared to home university course***Sliding scale from 1-5 (from Exceptional to Poor)**Sliding scale from 1-3 (from More Difficult to Less Difficult)ECQ

Questions: AcademicsExample III– Goal-based questions• Classes provided me with a greater depth of knowledge andunderstanding pertinent to my liberal arts education.Strongly agree / Agree / Not sure / Disagree• The program facilitated my ability to perceive the influence of cultureon myself and on others, and to interpret meaning from within acultural context.Strongly agree / Agree / Not sure / Disagree• The program motivated me to observe my own behavior and modifyit so that it becomes appropriate to culturally different situations.Strongly agree / Agree / Not sure / DisagreeGBQ

Questions: HousingExample I– Consumer-based questionsCBQ• Should we continue to use this housing situation?• Would you recommend that we retain this family in the future? Ifnot, why not?Example II– Ethnocentric questions• On a scale from one to ten, ten being the highest, how would yourank your host family?ECQ• What were the positive and negative aspects of your stay with thishost family?

Questions: HousingExample III– Goal-based questions• My homestay enhanced my learning about the culture• My homestay family included me in their daily life• My homestay family provided a forum for discussion and sharingStrongly agree / Agree / Not sure / DisagreeGBQ

6 Step Collective ExerciseBased on the provided program description: List the learning goals of each component ACTION VERBS! Echo the objectives in question formQuestions that assess to what extent the program succeeded in realizing itsprojected goals Identify the students’ adaptational challenges relative to thiscomponent Echo the challenges in question form List what the ideal student would do to achieve the targetedlearning goals Echo ideal student engagement in question form

Sample Program DescriptionSemester-long Hybrid ProgramMission StatementOffer motivated students an environment of authentic interculturallearning both inside and outside the classroomLearning Goals• Help students adapt to a variety of teaching and learning styles• Foster the building of meaningful relationships with host nationals• Optimize target language acquisition• Challenge students to become the co-creators of their experience abroad

Sample Program Components1. Individual homestay placement2. Community service option3. Membership in a local club or sport’s team4. Language pledge5. Courses taught by local faculty6. Direct enrollment option at local universityPresenter: Lilli EngleAmerican University Center of Provence, FranceFacilitators:Micheal Vande Berg, CIEE & Patricia Martin, SwarthmoreCollegeCIEE ConferencePhiladelphia, Nov. 2010

Benefits to the ProfessionThe process of alignment and accountability: Clearly defines program commitment and learning objectives Validates the level of adaptational challenge for the student Implicates student responsibility in the co-creation of theexperience and its learning outcomesPresenter: Lilli EngleAmerican University Center of Provence, FranceFacilitators:Micheal Vande Berg, CIEE & Patricia Martin, SwarthmoreCollegeCIEE ConferencePhiladelphia, Nov. 2010

In depthFor more regarding the outlined process of alignmentand accountability:Forum on Education Abroad, Standards of Good Practice on-line toolkitwww.forumea.org for a complete handbookUpcoming publication: Frontiers: The Interdisiplincary Journal of StudyAbroad, The Rewards of Designing Qualitative Assessment QuestionnairesAppropriate to Study Abroad, Fall 2011.Presenter: Lilli EngleAmerican University Center of Provence, FranceFacilitators:Micheal Vande Berg, CIEE & Patricia Martin, SwarthmoreCollegeCIEE ConferencePhiladelphia, Nov. 2010