ECN: Year 2 Evaluation Report

Evaluating Ourselves: What We Learned
A Summary of Findings from the Second Year of our Partnership Evaluation
(September 1, 2021 to August 31, 2022)
In 2019, the Theory of Change Working Group was tasked by the Steering Committee to develop a dynamic Theory of Change
framework (abbreviated illustration on the next page) as a foundation for evaluating the ECN's partnership engagement,
scholarship, and impact. This Theory of Change has guided the Strategic Learning Team in building and implementing an
evaluation plan. This report is a summary of evaluation findings from Year 2 of this evaluation.
Authors:
Jenae Gauthier, Anja Dzunic, Rich Janzen, Pieter de Vos, Pauline MacPherson
We would also like to acknowledge the individuals who were involved in the implementation of the evaluation:
Shelly Jun, and Hannah Bowley
ECN Partnership Evaluation Report 2021 | 1

Early Childhood Practices, Programs, and Policies
A Partnership
Theory of Change
Taking part in
our projects
leads to...
ENGAGEMENT
KNOWLEDGE &
SKILLS CREATED
ACTION &
IMPACT
LEGEND
Research
Questions
How will we
get there?
GUIDING
PRINCIPLES
1. Community Driven
2. Culturally Responsive
3. Equitable Participation
4. Action-oriented
OUR GOAL
To develop community-driven and
culturally responsive approaches for
producing and using evidence to improve
early childhood practice, programs, and
policy towards system effectiveness.
OUR OBJECTIVES
Objective #1
Deepen understanding
through engagement
and collaboration to
ensure that evaluations
are community- driven
and culturally
responsive.
How do we ensure that the demand
for, creation, and use of evidence are
community-driven and culturally
responsive?
Greater capacity
in ensuring that
research
collaboration is
communitydriven
Objective #2
Co-create and
implement research
and evaluation that
is mutually
beneficial to all ECN
partners.
Building authentic
partnerships and
collaborations
Greater capacity
in ensuring
equitable
participation of
stakeholders in
research
collaboration
Objective #3
Develop an Evaluation Hub
that connects people to
evaluation resources, learning,
and support to mobilizes
knowledge and build
evaluation capacity in the early
childhood field.
How do we increase knowledge,
skills, and motivation to create
& use meaningful evidence
through innovative capacity
building approaches?
Training and
mentoring
Greater
capacity for
ongoing colearning
Greater
capacity in
using evidence
Research and
evaluation
projects
Greater
capacity in
creating
quality/useful
evidence
Partners
ECN
Increased access to and use of
evaluation resources
Lasting connections
Deepened community of practice
How do we increase the value
and use of evidence for decision
making to improve practice,
programs, and policy?
Greater
mobilization of
people
Evaluation Field
Increased
understanding of
community-driven
and culturally
responsive evaluation
Knowledge
mobilization
Greater
mobilization of
evidence
Greater value
and use of
evidence in
decision-making
across
stakeholders
Early Years Sector
Increased culture of
learning and use of
evidence
Improved outcomes
in early childhood
Activities
Outcomes
ULTIMATE
OUTCOMES
More innovative early
childhood practices,
programs, and
policies
Greater system
effectiveness in
improving early
childhood outcomes

PARTNERSHIP EVALUATION: YEAR2
1
2
3
4
Purpose
The purpose of this evaluation is to:
Examine how we are working according to our
four guiding principles (community-driven, culturally
responsive, equitable participation, action-oriented).
Understand the impact we are collectively making.
This is in order to learn, grow, and adapt what we do (within the
partnership) and share lessons learned with external
stakeholders (beyond our partnership).
Evaluation Questions
To what extent are our partnership’s activities (engaging
communities, co-creating/co-learning, and EvalHub
capacity building) aligned to our four guiding principles
(community-driven, culturally responsive, equitable
participation, action-oriented)?
How, and to what extent, are we effectively building
capacity both internally as we equip ourselves to do our
work, and externally as we mobilize what we are learning
with others?
How, and to what extent, is our partnership’s work
reaching our ultimate goal to improve early childhood
practice, programs, and policy towards system
effectiveness? How are our collective efforts contributing
to the development of children in Canada?
What can we do to improve the effectiveness of our
partnership in meeting our desired outcomes? What are
we learning about how to carry out such a partnership?
Data Sources
The majority of data came
from this method!
Student Community of Practice Focus Groups
4 focus groups with 5-9 participants
5 open-ended questions discussed during monthly student CoP meetings
Data collection: January 2022 - July 2022
Meeting Minutes
Reviewed meeting minutes from all ECN working groups and committees to
triangulate themes
Data collection: September 2021 - August 2022
CUP/ECN Student Survey
Students responded to the survey before joining the ECN and at follow up
several months later
Data collection: September 2021 - August 2022 (n = 8 responses)
Post-Meeting Surveys
Responses were submitted online after Knowledge Exchange Working Group
and Steering Committee meetings
Data collection: September 2021 -
August 2022 (n = 16 responses)
Activity Tracking
ECN activities and outputs were tracked
Data collection: April 2019 - August 2022
Role and work sector of postmeeting
survey participants
Researcher (40.91%) Leader/Manager (13.64%)
Funder (4.55%) Evaluator (13.64%)
Student (18.18%) Intern (4.55%)
Community Member (4.55%)
ECN Partnership Evaluation Report 2021 | 3

Evaluation question 1: To what extent are our partnership's activities (engaging communities, co-creating/co-learning, and EvalHub capacity
building) aligned to our four guiding principles (community-driven, culturally responsive, equitable participation, action-oriented)?
Students expressed that there is lack of clarity in the definitions of the four guiding principles. They perceived the values as
contextual, i.e. the values will mean different things in different settings and projects. Students in particular didn't know how to
measure these values and thus felt it is difficult to gauge to what extent the ECN is living them out. Nonetheless, partners and
students identified some aspects of the ECN that align with the principles.
Community-driven
Our partnerships activities and
projects are considered to be
community-driven to different
extents, depending on the
project at hand. The ECN is
doing well in seeking
engagement with
communities, through for
example surveys and
engagement sessions, to
understand their needs, assets
and inform decision making.
I'm seeing these values as
more of like a mixing board.
Because we can't really have
100% of any of them because
of capacity, but I think they
keep us grounded.
Culturally
Responsive
Efforts to be more culturally
responsive have been made
within the network over the
past year, primarily by
incorporating it into capacity
building initiatives.
Indigenous
component
in UEval
CR
Workshop
representative
samples
teaching
CR-E in
lectures
However, some students
commented that key
stakeholders they are working
with are not always culturally
diverse, despite the fact that
front line workers represent a
variety of cultures.
Equitable
Participation
Internally, the ECN runs in an
equitable manner. A diversity
of perspectives is represented
in the partnership.
Externally, the ECN is seeking
engagement with a variety of
people and sectors to inform
forthcoming activities.
Partners and students also
perceived the ECN's capacity
building initiatives as lowbarrier
as
they are accessible
nationwide
tuition subsidies are
available
Action-Oriented
Some partners felt that
meetings are well organized,
while others felt they should
have clearer outcomes and be
more focused.
Students seemed to define
action-oriented as seeing
results or an impact from the
work they are doing. This was
difficult to measure for them.
Students expressed higher
confidence of their projects
being action-oriented when
projects had high levels of
community engagement.
How is this gonna actually
translate into action on the
ground for practitioners?
ECN Partnership Evaluation Report 2022 | 4

Evaluation question 2: How and to what extent are we effectively building capacity both internally as we equip ourselves to do our work,
and externally as we mobilize what we are learning with others?
Internally
Students identified many ways during the focus groups through which they are building their own capacity.
These are the skills students mentioned to have gained most often:
Evaluation and research knowledge and skills, such as
soft skills, i.e. communication, leadership, being more
reflective, collaboration and networking
hard skills, i.e. data analysis skills, academic writing,
literature review, knowledge mobilization and translation
Understanding of community-based collaboration, including
the value of community-driven work and working
collaboratively with stakeholders
the fluidity and iterative nature of community-based work
being proactive and anticipating concerns
building relationships and reaching out for help
Group facilitation and public speaking skills, including
running focus groups
mentoring students
facilitating meetings
planning what questions to ask and how to ask them
developing presentations
better prepare them for future
employment, continuing
education, and
thesis work
help them navigate
different contexts
and community
partners
Students explained how these
skills will:
help them do
community-based
participatory
research and
evaluation
teach them
independence in
work responsibilities
Students in particular mentioned that UEval/QEval
was helpful in developing their confidence in
facilitation skills, as well as navigating group dynamics.
Some students were confident that the skills will be used
somewhere but they were unsure how exactly or where they
will be able to apply them.
ECN Partnership Evaluation Report 2022 | 5

Evaluation question 2: How and to what extent are we effectively building capacity both internally as we equip ourselves to do our work,
and externally as we mobilize what we are learning with others?
In the CUP/ECN student survey, students (n = 8) reported to
have improved in many research and evaluation skills. Results show
statistically significant improvement in the following six skills:
it's overwhelming
sometimes to just be soaking in so
much information and practicing a
lot of new skills in some areas. But
it's really, really rewarding at the
same time.
There were some areas where students have not improved
their skills, including: teaching, EDI, and creating
project/community reports. Students' were also asked to rate
their knowledge and confidence in various research and
evaluation areas. Findings showed an increase in student's
knowledge and confidence as a result of their experience with
the ECN.
*
*
*
**
***
Note: Students rated their skills/competence on a 5-point likert scale - novice (0) to advanced (5). The chart
shows the mean of all responses with a statistically significant change from pre-survey to follow-up.
*p = .049; **p = .011; ***p = .033
Capacity Building is a
process, not an outcome.
Through the work with the ECN, students’ attitude,
perception, and understanding of capacity building
has changed.
Even just experiencing
those, like different group
dynamics, is really helpful in
terms of future work,
right? Because it's not like
I'm going to be working
independently forever. A lot
of like research and
evaluation involves
collaborative
group work.
ECN Partnership Evaluation Report 2022 | 6

Evaluation question 2: How and to what extent are we effectively building capacity both internally as we equip ourselves to do our work,
and externally as we mobilize what we are learning with others?
Externally
Partners and students also described how the ECN is building capacity externally, for example in communities. Activity
Tracking allowed us to identify the activities we are engaging in to build capacity in communities. For a full list of
partnership activities, go to page 4-5 and 9-10 of the partnership report.
292
Course
learners
3
experiential
living labs
75
community
organizations
supported in
creating evaluation
plans
305
resources in
the ECN's
Repository
Experiential
learning courses at
4
partner
Universities
building capacity of communities and
meeting their needs through courses
The ECN is...
Recent projects include:
Language revitalization
in Indigenous
communities
Early learning and
childcare for newcomer
families
Supportive housing
models for teen
families
Building capacity for
Canadian child and
youth mentoring
organizations
mobilizing and facilitating
access to evaluation resources
involved in and supports
numerous evaluation projects
ECN Partnership Evaluation Report 2022 | 7

Evaluation question 3: How and to what extent is our partnership's work reaching our ultimate goal to improve early childhood practice,
programs, and policy towards system effectiveness? How are our collective efforts contributing to the development of children in Canada?
Students identified how their projects are contributing to
our partnership objectives.
Partners and students expressed how the ECN is:
Research Projects
with MRU
Experiental Learning
Evaluation
Objective 1
Deepen understanding through
engagement and collaboration to
ensure that evaluations are
community- driven and
culturally responsive.
Lunch and Learns
Knowledge Exchange
Working Group
Increasing capacity
in the ECD sector
through research and
evaluation projects,
but the extent of it is
unknown.
Creating
collaboration
opportunities to
inform decisionmaking.
AVIRT
Objective 2
Co-create and
implement research
and evaluation that is
mutually beneficial to
all ECN partners.
All In For Youth
UEval/QEval
Alberta Mentoring
Partnership
CD-E Scoping Review
Objective 3
Develop an Evaluation Hub that
connects people to evaluation
resources, learning, and support
to mobilizes knowledge and
build evaluation capacity in the
early childhood field.
Note: This is not a complete list of projects the ECN is supporting, but includes only those that were
reflected upon in student focus groups.
Mobilizing resources
and evidence
through the ECN
repository,
webinars and other
methods.
While
the repository is a
good resource,
it should be made
more user
friendly.
Looking back at our Theory of Change, we are meeting several
of our outcomes, but as expected have not yet reached our ultimate
outcome of creating more innovative ECD practices, programs, and
policies towards greater system effectiveness.
ECN Partnership Evaluation Report 2022 | 8

Evaluation question 4: What can we do to improve the effectiveness of our partnership in meeting our desired outcomes? What are we
learning about how to carry out such a partnership?
How to improve...
The Student Experience
How to improve...
Meetings
In student focus groups and the student survey, students identified three
key areas of improvement:
1) Orientation and Understanding the ECN
Newer students received an orientation to the ECN and
perceived it as helpful
Booklet of ECN acronyms may be needed
Assigning a student mentor to incoming students
Students wish to understand better how they fit into the
bigger ECN partnership
2) Internal Relationships and Communication
Students wanted to have time to ask questions to
leadership, regular team meetings (ECN started bi-weekly
team meetings recently), and supervisor check ins
ECN team text group chat and someone who moderates it
'Buddying' up on projects
Creating deeper connections with co-workers
Continuing CoP as a safe place for students to check in
3) Projects and Skills
Providing opportunities to work on various
tasks/projects and gain a range of new skills
Following up with students to ensure their needs
and interests are met
I had a whole
week kind of just to
focus on getting to know
what ECN is doing. So that
was, yes, it was a lot of
information. But it was
helpful to kind of get my
foot into it easy and
at least start understanding
what it's about and
what projects
it has.
I think some deeper
connection would be
really nice to see
because I think that
would also really
expand our capacity to
support each other in
the more evaluationbased
life that we're
living with ECN as
well.
Following suggestions were given
by meeting participants to
improve meetings:
Use collaborative platforms
to increase engagement
Increase meeting efficiency
(e.g., shorter meetings, send
out documents to review
beforehand, establish clear
outcomes)
Encourage everyone in the
meeting to contribute to
discussions
How can we get more
people to contribute to this
discussion?
ECN Partnership Evaluation Report 2022 | 9

Evaluation question 4: What can we do to improve the effectiveness of our partnership in meeting our desired outcomes? What are we
learning about how to carry out such a partnership?
How to improve...
The Wider Network
Suggestions were also provided by partners and students on how to improve the partnership in general.
More
Intentionality
is needed in
targeting the desired audience
leveraging partners to grow the network
building communities of practice
reaching out to other sectors
ECN's
ECB Initiatives
need to
be tailored to the needs and resources of specific community organizations
provide support for implementing evaluations
incorporate an EDI lens
The ECN
could
improve
Communication
& Enagement
by
using diverse Knowledge Mobilization techniques
using audience-appropriate language and promoting a learning lens to evaluation
increasing awareness of the ECN
improving information exchange between our partners
seeking new perspectives
building relationships with Indigenous evaluators
ECN Partnership Evaluation Report 2022 | 10

Conclusion
In the second year of our partnership evaluation...
What have we done well?
We have continued to emphasize our four guiding
principles and to work towards meeting our objectives and
outcomes, including mobilizing evidence, greater capacity
for collaborative research, and equitable participation. We
are taking steps towards meeting our ultimate goal!
Students have gained numerous skills which they
will be able to use in other settings as well.
While continuing to build internal capacity, we have
shifted the focus to building capacity of communities.
What are some areas for growth?
Continue to promote
engagement and deeper
connections, especially among
students and partners.
Be more intentional and strategic in
building capacity of communities,
network growth, and internal and
external communication.
Reach a shared understanding of the
four guiding principles: communitydriven,
culturally responsive, equitable
participation, and action-oriented.
ECN Partnership Evaluation Report 2022 | 11