Watershed Achievements Report

Statewide Watershed Achievements Report 2015
Social Indicators for Nonpoint Source
Pollution in Minnesota
The EPA Region 5 funded a multi-state effort to create a
social evaluation framework, called the Social Indicators
Planning and Evaluation System (SIPES). Minnesota
joined this effort in 2006, testing the SIPES framework
with four urban and rural NPS projects. The results of
the testing phase of the SIPES framework was that this
evaluation tool did not meet local needs. Some partners
declined to use SIPES again for post-project summative
evaluation.
Our research found that Minnesota’s NPS projects vary
considerably by type, scale, and capacity to evaluate
outcomes. About half of Minnesota’s projects have
audiences too small for random probability sampling as
required by SIPES. Local partners reported that SIPES was
too complicated and costly for them to use. Minnesota’s
NPS audiences are also more diverse than other states,
which is not captured by the SIPES framework.
Therefore, we adapted and tested an alternative method
called the knowledge, attitudes and practices (KAP) study
as part of our approved work plan. The KAP method
worked well in instances where audiences are too small
for SIPES, or where standardized SIPES questionnaires
are not a good fit with the characteristics of the local
audience. By April 2013 twenty-five KAP studies were
completed or in process, mostly commissioned by
local watershed agencies. The KAP method is useful for
formative and summative evaluation, and is helpful in
organizing civic engagement and education strategies.
We have completed a training package for local partners
on social research and evaluation methods, including a
protocol for the KAP study method.
In summary, the social indicators project was completed
on time and within budget, and yielded useful findings
and practical tools for watershed projects. Demand for
KAP studies has grown in Minnesota. SIPES continues to
be used by other Region 5 states.
Goals
• The overarching goal of this project was to create
practical evaluation methods and tools for the
staff of 319-funded NPS projects that can measure
changes in human behaviors, knowledge and
capacities, as measured through pre (formative)
and post (summative) project evaluation methods.
Develop design guidance material based on results
of experimental work at SWROC and modeling done
using BASIN.
• Provide 319-funded NPS projects with the SIPES/
social indicators data management and analysis
(SIDMA) framework, and train project staff in its use
and applications. This indicator-based evaluation
framework is intended for 319-funded projects with
larger target audiences, and enables evaluation data
to be compared across states and within EPA Region
5. The SIPES and SIDMA tools may eventually be
required by EPA for use in all 319-funded projects.
• Provide smaller-scale NPS projects in Minnesota
with practical “supplemental” evaluation tools and
methods to determine project impacts and outcomes.
Supplemental tools are intended for situations where
random sampling and standardized questionnaires
are not appropriate.
Results that count
The social indicators framework (SIPES and SIDMA)
was tested in four diverse cases in Minnesota (a midsize
urban stormwater project in Duluth; a municipal
winter maintenance training program to reduce chloride
application by snowplow drivers; a small rural agricultural
project in southwestern Minnesota; and a large-scale
random probability survey of producers in four counties
in the Buffalo-Red watershed). In other Region 5
states, members of the regional social indicators team
generally conducted the surveys for the local partner. In
Minnesota, we asked the local partner to actually conduct
and implement the SIPES surveys, since they were the
intended end-users. The testing process found that
the framework was too burdensome for local partners
(generally municipalities, counties, and SWCDs).
In response to their feedback, we adapted and tested
the KAP study method for use in NPS projects at
different scales. In all cases the KAP method has worked
very well, yielding information that has contributed
to engagement/outreach strategies and educational
content, as well as the original intent (formative and
summative evaluation). The KAP method has proven
to yield valuable information for project staff that has
enabled them to fine-tune their outreach efforts and
to customize educational messages. With such direct
and immediate information about pre/post project
Minnesota Pollution Control Agency www.pca.state.mn.us 38