Storytelling Approaches to Program Evaluation - The California ...

NOVEMBER2007Storytelling Approachesto Program Evaluation:An IntroductionPICTURES WRITING THEATERFunded by:

This document provides apractical introduction to storytellingapproaches being successfully appliedin grassroots program evaluation.These approaches can be used in almost any setting, with any numberof participants, in any language. They are flexible and can be creativelytailored to the needs and styles of your particular organization or project.PICTURESWRITINGTHEATERAdapted by Sylvia Sukop from an original report by Joseph Tobin and Gustavo E. Fischman.

Table of ContentsIntroduction 2Gathering Stories: Basic Interviewing Approaches 2Story Circle 3Storytelling Interviews 3Oral History Interviews 4Institutional Memory 4Getting Creative: Documenting Stories in Pictures, Writing and Theater 4Visual Documentation 4Scrapbooking and Story-quilting 5Story Theater 5Case Studies 6Vignettes: Short Stories as Illustration 6Five Tips for Gathering and Organizing Stories 6Evaluation and Beyond: Using the Stories you Collect 7Public Relations and Fund Raising 7Policy Advocacy 8

There is no “one-size-fits-all”approach to program evaluation.Evaluating the impact of programs aimedat social change and involving diverseconstituents demands especially innovativeand culturally appropriate approaches.Storytelling is proving to bea highly effective, culturallyappropriate approach tograssroots program evaluationthat gives voice to individualand collective experience.IntroductionTraditional evaluation methodologies–whichrely on formal questionnaires and statisticaldata that can be scientifically analyzed–failto give program participants a chance to telltheir own stories in their own ways.Storytelling is an ancient tradition thathas, over time and across cultures, servedmany different purposes, from educationand the transmission of values topolitical mobilization and pureentertainment. It can takemany forms, from oral andwritten narratives to gesture,movement, art, music,movies and more.Storytelling is a powerful mode ofhuman expression and a sophisticatedform of “meaning-making.” It beginswith a storyteller, a singular experience,a unique point of view.This is one reason that storytelling hasbecome such a useful tool in programevaluation: It accommodates diverse voicesand perspectives, while making the most ofthe particular resources and ways of learningreadily available in your program. Unlike atraditional evaluation approach that isimposed from outside, the storytellingapproach emerges organically from within yourown organization, projects and participants.Gathering Stories: BasicInterviewing ApproachesThese basic approaches use a simpleinterview or question-and-answer format.There should always be a clearly designatedinterviewer or facilitator guiding theconversation. Ideally the conversation willbe recorded (audio or video) or detailednotes will be taken. Using these recordingsand notes, you will be able to share thestories you have gathered with others. Before2STORYTELLING APPROACHES TO PROGRAM EVALUATION: AN INTRODUCTION

Why Storytelling Works• Storytelling values and respects diverse ways of knowing and learning.It is empowering and participatory, and is based on popular knowledge.• Stories can be used effectively alongside statistics and surveys.Including stories in your program evaluations puts a face on the facts and figures,and helps you figure out what’s working, what’s not, and why.• Stories speak to a broad audience.Including your stakeholders’ voices and perspectives can help you communicateto your stakeholders, your funders, and the larger community what you areaccomplishing and why your program is so important.you get started, be sure to read “Tips forGathering and Organizing Stories” (page 6).Story CircleAt the end of a staff meeting or sessiongo around the group in a circle asking foreach participant to share a story. To getthe conversation going, the intervieweror facilitator can ask an open-endedquestion like, “What have you learnedfrom being part of this program?” or“How has participation in this programchanged your life?”At the end of the session ask the groupto summarize the most important lessonslearned based on the stories told in the circle,and be sure to record or write these down.You can implement the story circle everythree to six months to assess your programprogress and involve diverse stakeholders.Storytelling InterviewsEveryone has a story to tell, but manypeople need help to get their stories out.One or more staff members can sit downwith program participants one-on-one orin small groups to gather their stories.The interviewer can begin by explainingthe purpose of the meeting, by sayingsomething like, “Our agency wants tohear from our participants abouthow our programs are helpingthem. Hearing your experiencewill help us know how we are doingand give us ideas for improvement.”The interviewer shouldask questions to get theperson talking, but notinterrupt unnecessarily.The interviewer can askthe participant to describe:Including stories inyour program evaluationsputs a face on the factsand figures, and helps youfigure out what’s working,what’s not and why.• The problems he/she is facing thatled him/her to come to the agency;GATHERING STORIES: BASIC INTERVIEWING APPROACHES 3

See “Tell Us YourStory” HandoutThis handout can be used toguide your interview and evenshared with participants inadvance, if appropriate.• The activities in whichhe/she has participatedor the services he/shehas been receiving;• How these servicesare making a differencein his/her life.The interviews should be taped and thentranscribed and edited, cutting out commentsthat are off topic and focusing on the mostcompelling stories told by the participants.Oral History InterviewsIncluding the voices of the people in acommunity who have familiarity with itshistory is a way to gather information onhow it has changed over time. It can help todescribe the historical, social and culturalcontext of your program as well as how thecommunity and the program have evolvedover time. In addition, it can help youidentify community cultural events thatexisted in the past that can becomeresources for the future.Oral history interviews may also be usedto collect information on the lives of yourstakeholders before you initiate your programactivities and then after your program hasconcluded, to see how their livesmay have changed as a resultof their participation.Institutional MemoryCollecting and cataloguingstories not only fromparticipants but also fromstaff can be an excellent wayof recording the history and charting theprogress of a program and the growth of anorganization. For example, staff memberscan be asked to tell stories about how theprogram has evolved over time. Thesestories can be recorded and transcribed.This process can provide program staff agrowing archive of stories that can beanalyzed to chart growth in individualprogram participants, staff members, theprogram and the organization as a whole.Getting Creative:Documenting Storiesin Pictures, Writingand TheaterVisual DocumentationStories can be told in pictures as well aswords. Program staff and participants canbe given cameras and invited to create avisual record of their participation in theprogram. These photos can be organizedinto a montage on a poster or presented ina digital slide show to demonstrate programactivities and growth. This participatoryevaluation approach is called “Photovoice.”It utilizes photographs taken by programstakeholders to enhance need assessments,discussions and reflection, gather data,promote dialogue, conduct participatoryevaluations and communicate results withvarious audiences, including policymakers.Similar approaches can be done with videocameras, and when shared through computers,the approach is called “Digital Storytelling.”With participants’ permission, it is bestto save the stories as digital files. Stories4STORYTELLING APPROACHES TO PROGRAM EVALUATION: AN INTRODUCTION

can then be selected to depict specificprogram impacts, successes and challenges,including diverse views and perspectiveson how the program has impacted the livesof involved stakeholders and, possibly, thelives of their communities.Scrapbooking and Story-quiltingCollecting artifacts and images and placingthem in an album to document programactivities, process and outcomes is a veryold practice that has recently become apopular hobby. Participants and staff can beasked to keep a scrapbook throughout theirparticipation in a program to document theiractivities, challenges, accomplishmentsand growth. These scrapbooks can thenbe used as a form of portfolio assessmentin program evaluation.Quilting is another folk craft that can beused for telling stories. For example, in aquilt-making activity, each participant cancontribute a square that contains a personalreflection on the program, its process andoutcomes. The participants can discuss howto organize their squares to make a quiltthat tells the story of their individual andcollective experience. Story-quilting canbe used to assess and promote communitypride, by describing how the community hascome together to accomplish activities andmake changes that resulted in improvedcommunity life and a sense of belonging.Story TheaterTheater has been used effectively in the fieldsof health and mental health services and insocial movements. Theater can dramatizecommunity concerns and demonstratepossible solutions by ordinary people whoare trained neither as playwrights nor actors.Community members are asked to developstories that they turn into scripts andperform, either for themselves or for thepublic. Most often, the script emergesfrom role-play exercises.Elements of story theater can be adapted intoprogram evaluation. For example, every monthor two, you can ask your program participantsto role-play the process, challenges andaccomplishments, as well as outcomes ofa program. If you tape the role plays (audioor video) over the course of a year, theyprovide a running record of growth.Story theater can also be anexcellent means to inform thecommunity about your programevaluation results. For example,an organization in Californiacalled Lideres Campesinas(Farmworker Women Leaders)writes short plays in whichthey describe the livesof women who haveexperienced domesticviolence and howtheir lives changed asa result of becomingactive members ofthe organization.Online ResourcesPhotovoicewww.photovoice.comThe Center forDigital Storytellingwww.storycenter.orgGETTING CREATIVE: DOCUMENTING STORIES IN PICTURES, WRITING AND THEATER 5

Vignettes are strongpersonal statementsthat give life–a humanface–to collected data.Case StudiesCase studies are more in-depth,written stories of the livesof your stakeholders. Theymay also depict the placeswhere they live, where yourorganization is located, and howcommunity life and other factorssuch as the environment and the socialdeterminants of health (e.g., employment,race and housing) interact with the processesand outcomes that emerge as a result ofyour stakeholders’ participation in yourprogram. A case study may include:• Program participants’ biographicalinformation;• Their reasons for being involved;• The services they received or theactivities they participated in; and• Challenges, successes and outcomes.Vignettes: Short Storiesas IllustrationStories are used to illustrate andcontextualize the points you makewith facts and numbers. For example,after presenting statistical informationon the activities that you implementedin the past year, you can add a box thatcontains an account told by a staff memberor program participant that describes aparticular event that supports the statisticaldata. Vignettes are strong personal statementsthat give life–a human face–to collected data.Five Tips for Gatheringand Organizing Stories1. Be consistent and systematic.Like keeping a diary, you need to have aplace to record your stories; you need toallow time to write them down and youneed to keep at it. Do a little at a timerather than waiting until a lot of timehas passed and trying to remember andrecord weeks or months of stories all atonce. It’s easier to record stories whenthey are fresh in your mind than to goback and re-construct them.2. Designate Storytellersand Story-collectors.Everyone has a story to tell, but noteveryone is a good storyteller. Identifythe good storytellers among your staffand program participants and encouragethem to tell their stories and also tohelp other, less skilled storytellers, gettheir stories out. You also need toidentify someone who takes primaryresponsibility for story-collecting.By story-collecting, we mean recording,writing and transcribing stories aboutyour program. This takes time and energy.You may want to train a staff memberto be the designated story-collectorfor your agency or you may want torotate the responsibility each monthor for each event.6STORYTELLING APPROACHES TO PROGRAM EVALUATION: AN INTRODUCTION

3. Be strategic in your choice of storiesyou include in a program evaluation.Everybody has a story to tell, but thisdoes not mean that every story told toyou by staff members and programparticipants needs to end up in yourreport. Strategically select the storiesthat are most appropriate for the taskat hand—and these may not alwaysbe success stories! Certain stories mayillustrate problems and challengesencountered by your stakeholders;others may demonstrate your program’ssuccessful interventions or barriers tosuccess; and still others may dramatizethe particular needs, assets and resourcesof the communities you serve.4. Stories do not tell “the whole story.”It is important that you complement thestories you include in your evaluationreport with other sources of information.Stories should be combined with surveys,focus groups, observations and othermethods of evaluation. Providingmultiple forms of data and includingthe perspectives of the full range ofyour participants will enhance thequality of your program evaluationas well as the your stories’ impact.5. Consider the ethical implications ofsharing personal and institutionalstories. Always ask participants forpermission before recording theirstories. Also get their approval toshare with others the stories they tellyou, explaining that their real namewill not be used in connection withtheir stories. It is good practice to havewritten consent (see sample consent formincluded with this document) to assureconfidentiality in the use of personalnarratives; and to apply responsible andethical research practices, ensuring thatthe human rights, dignity and welfareof human subjects are protected.Evaluation and Beyond:Using the Stories you CollectYour participants’ stories can serve as apowerful tool in service of your organization’sgoals. Think about the particular audiencethat you want to target, from fundersand policymakers to the media andthe general public.Public Relationsand Fund RaisingPrograms can use storytellingfor publicity, fund raising andrecruitment of new participants,stakeholders, volunteersand staff. The same storiesyou use for evaluationcan be used to explainto potential participants,other community serviceorganizations, potential fundersand the mediawho you are, whatyou do, what youhave accomplished,successes and challenges,and what you need.It is good practice tohave written consent.See Sample ConsentForm Included inthis Brochure.EVALUATION AND BEYOND: USING THE STORIES YOU COLLECT 7

Policy AdvocacyCompelling humanstories will strengthenyour policy advocacymessage. Stories of yourparticipants’ struggles and howyour program helped to mitigate them canbe used to raise awareness of your programimpacts. Stories of their resiliency and ofthe gains they make in your program canbe used not just for funding but alsofor social and systemic change.How to Disseminate Your StoriesSelect statements from your stories that “speak to” specific audiences and presentthem in the appropriate format for that audience.• For funders: Write an evaluation report with an executive summary that highlights,in brief, the main stories compiled in your report.• For policymakers: Write policy briefs that incorporate vignettes of your mostcompelling stories along with salient facts and figures.• For the media: Write a press release that includes one or more compelling storiesand includes direct quotes from participants.• For community members and stakeholders: Weave stories and quotes into yourorganization’s publications such as newsletters, brochures and annual reports. Writean article in a popular community newspaper or a community newsletter. Use storytheater to dramatize community concerns and potential solutions.8STORYTELLING APPROACHES TO PROGRAM EVALUATION: AN INTRODUCTION

Tell Us Your StoryHearing your experience will help us know howwe are doing and give us ideas for improvement.1First, tell us a little about yourself.2Please describe the problem or problems you were facing that first led you to come to us.3Describe some of the activities in which you have participated or the services you havebeen receiving.4Have these activities or services made a difference in your life?5Looking ahead, what are some of the new or continuing challenges that you expect toface? Do you plan to continue participating in our programs?

Consent FormWe would like to assess the impact of your participation in our activities and services bymaking audio or video recordings of your stories. Through this assessment, our organizationwill learn what worked and what did not work and why. This learning will help us improveour programs and services.We may also use the products of our assessments to share the stories we collect with ourcommunity and other stakeholders or to advocate for our clients’ needs. We may use theseproducts to publish our accomplishments and to seek support for our ongoing services,program(s), and/or activities. We may publish information, stories, photographs and artworkthrough various media—including but not limited to print, electronic and audio-videorecordings. Examples of these publications may include but are not limited to newsletters,brochures, reports, Web sites, slideshows, PowerPoint presentations, program photo albumsand/or audio-visual public service announcements.No media shall be used for exploitation or promotion of activities unrelated to the missionof our organization, .Please complete and return this form to our office. In order to register for our program, theform must be completed, signed and returned regardless of your decision.Your Name/Youth’s NameDate of Birth / / (month/date/year) Today’s Date / /I GIVEpermission to publish my name/child’s name, image, written work and/or artwork forthe purposes stated above.(organization’s name)I DO NOT GIVE permission to publish my name/child’s name, image, written workand/or artwork for the purposes stated above.SignatureProgram Participant or Parent/GuardianPrint NameProgram Participant or Parent/GuardianOffice Use Only:

STORYTELLING APPROACHES TO PROGRAM EVALUATION: AN INTRODUCTIONEvaluating the impact of programs aimed at social changeand involving diverse constituents demands especiallyinnovative and culturally appropriate approaches.

Evaluateyour programsby storytelling.Everyone has a storyto tell, but many peopleneed help to gettheir stories out.1000 North Alameda StreetLos Angeles, CA 90012800.449.4149www.calendow.org© 2007 The California EndowmentCPA/StorytellingTCE 1019-07