Terms of Reference CPAP Moldova Evaluation - unfpa

Strategic plans, and how UNFPA increased aid effectiveness and contributed to United Nationscoherence.Evaluation Questions:• Are the project strategies adapted to the environment in which it operates, in line with nationalneeds, policies and priorities?• Are there synergies or complementarities between the development actors, both government andUN entities?• How successfully has the country office engaged in the UN reform process, strengtheningcooperation with other UN Agencies through joint initiatives?• To what extent did the joint programme modality contribute to the achievement of CountryProgramme results?• What are the areas of UNFPA’s comparative advantage that the country office needs to focus onin the next Country Programme of support to Moldova and within the context of UNDAF and therapidly evolving aid environment in the country?Effectiveness:The assessment of effectiveness considers or examines the extent to which the CP has achieved itsplanned results, including outputs and outcomes and the extent to which the achievement of outputs andoutcomes were as a consequence of UNFPA assistance and support other than other interventions andfactors.Objectives:• Evaluate the extent to which planned results, including agreed outputs and outcomes, have beeneffectively achieved as result of programme implementationEvaluation Questions:• As per quantitative/qualitative targets of the Country Programme Action Plan, to what extenthave knowledge and skills on sexual and reproductive health among students and vulnerableyoung people effectively improved due to support provided by the programme, such asinstitutionalized peer-to-peer education and special education for vulnerable group?• To what extent have availability of quality reproductive health information and services as well asSTI/HIV infection prevention and care services for young people and vulnerable groupseffectively increased?• To what extent has the Programme effectively strengthened the mechanism of quality assuranceand commodity security?• How effectively has demographic research been used to support the Programme implementation?4

• Did the programme activities increase the capacity of local NGOs to develop and implementprogrammes on gender-based discrimination and violence prevention?• How have human rights and gender equality considerations been effectively mainstreamedthroughout Country Programme development and implementation? Has the programme designbeen appropriate for a sustainable mainstreaming of human rights and gender equalityconsiderations throughout its implementation?EfficiencyThe assessment of efficiency considers or examines how economically and optimally inputs of thetechnical assistance (financial, human, technical and material resources) have been used to produceoutputs. The assessment of efficiency attempts link outputs to resources expended and assesses whetherthis happened as economically and as feasible as possible and the extent to which the quantity and qualityof the results justify the quantity and quality of the means used for achieving them and whether they wereachieved on time. In the implementation of the CPAP, assess how efficient the inputs and resources wereutilized to produce the results or outputs in respect of the following.Objectives:• Evaluate how economically or optimally financial, human and technical inputs have been used toproduce outputsEvaluation Questions:• How efficiently has result based management been applied in achieving country programmeresults?• Were inputs provided and outputs met on a timely basis?Impact• Assessment of the impact of the project considers the longer term and ultimate results attributableto the CPAP interventions that were being implemented. It considers the positive and negativelong term effects, which may be economic, socio-cultural, institutional, technological andenvironmental or other effects. Thus in the evaluation of the interventions, the assessmentshould determine the positive and negative changes produced by the intervention directly orindirectly, intended or unintended. Assess the extent to which the assistance has made a realdifference to the beneficiaries.Evaluation Questions:Do the beneficiaries and other stakeholders affected by the programme perceived the effect of theprogramme interventions on themselves?What are the perceptions of the different stakeholders, especially the Government of Moldova,implementing partners and other United Nations organizations about the overall impact ofUNFPA program?5

Leadership, Management and Institutional Arrangements• Leadership, management and institutional arrangements considers the governance structure of theassistance, the leadership and management of the interventions by UNFPA in terms of technicaladvice, coordination, consultations, reporting, support and backstopping, funding modality andarrangements. How effectively and efficiently has UNFPA technical assistance facilitated theabove and ensured relevance, impact and sustainability of the project.1. How efficient have the processes and systems followed been including the application of resultsbased management (RBM) in achieving country programme results? Assess effectiveness ofcountry office programme monitoring and information system?2. To what extent has the joint programme modality (in the context of Delivering as One) contributedto achievement of Country Programme results?3. Is there effective coordination among the government, UNFPA and other implementing partners?4. STAKEHOLDER PARTICIPATIONThe success of the evaluation hinges on a full stakeholder consultation and participatory evaluation thatprovides meaningful participation of all development interventions partners, beneficiaries of the nationalprogramme and other interested parties. Stakeholder participation, through a participatory workshopapproach, forms a critical component of the evaluation design and planning, information collection,documentation of findings and development of the evaluation report and dissemination of evaluationresults A stakeholder consultation and participation list is attached as Annex 1. Main stakeholders includeUNFPA Country Office and Regional Office staff, UN Agencies, Development Partners, Implementingpartners (state and non-state actors, including CSOs), and beneficiaries.5. EVALUATION METHODOLOGYThe evaluation will be conducted in close coordination with the UNFPA Moldova Country Office. It issuggested the Country Programme evaluation team is composed of three evaluators covering the threeProgramme components. Two national and one international consultant will be recruited. Theinternational consultant will also act as Team Leader.The evaluation methodology framework is an overarching guide. The evaluators should use it tostrengthen/develop/adapt an implementation design based on the understanding of the programme andwhat is being evaluated. In particular, evaluators are expected to submit a context specific evaluationstructure/outline as part of the inception report.The evaluation structure/outline shall be devised in conformity with UNFPA evaluation guidelines andthe norms and standards of the UN Evaluation Group, and might include the following components:i. Desk review of Government and UN policies, programs, relevant UNFPA programme documentsfor the three CP components, programme materials developed such as the training manual,brochures and other outputs, workshop reports, progress reports, donor reports, evaluation reportsand other relevant publications;6

ii.iii.iv.Carrying out individual or collective interviews with partners and stakeholders listed in thestakeholder consultation and participation list;Organizing focus group discussions with project stakeholders and beneficiaries to determine paceand performance of the implementation of activities and possible results of these activities as perthe project outcome and output indicators;In the absence of midline and endline data, use available data collection methods as much aspracticable to collate feedback on outcome of the project from beneficiaries and project partners.The evaluators will undertake triangulation and consolidation of the data obtained from themethodological processes outlined in the evaluation methodology framework. They will also alignobjectives or outputs of the CP and evaluation to the specific questions, performance indicators, datasources and data collection requirements to facilitate the evaluation proper.In accordance with Department of Oversight (DOS) evaluation quality standards (Annex 2), theEvaluation Team should also discuss how it would control bias and/or acknowledge limitations due touncontrolled bias, in the absence of control groups or counterfactuals.Key recommended stepsPreparation• This involves briefing, orienting and clarifying relevant issues and approaches to the evaluators aboutthe key concepts including delivering as one approach, SWAp processes and policy dialogue.Inception• The evaluation team review existing documents, meet the UNFPA CO and prepare a brief inceptionreport no more than 5 days from the day of being contracted. The inception report will include a clear,time and resource bound detailed work plan, clear evaluation questions and tools. An inception report(Inception report format Annex 3) to be prepared by consultants and presented to UNFPA CO andEvaluation Management Committee for discussion and agreement before the evaluation team movesto the implementation phase.Data collection• Desk reviews and analysis of key and relevant documents including implementing partners AWPs,progress reports, reviews, assessment reports etc. The list of key documents is attached Annex 4.• Interviews with key informants, UNFPA staff, UN agencies, selected Development Partners,Implementing partners (state and non-state actors including CSOs) and beneficiaries.Analysis• Evaluators will analyse collected data/information, seek clarifications from different sources to enrichthe findings.Reporting• Evaluators will present and submit report according to the agreed format to the UNFPA Management.7

Dissemination• Dissemination of the findings to the appropriate stakeholders and beneficiaries will be done as peragreed schedule.6. ETHICSThe evaluation of the project will be conducted along highest ethical and legal standards and with dueregard for the welfare of those involved in the evaluation, especially women, children and members ofother vulnerable or disadvantaged groups and in accordance with the UNEG’s Ethical guidelines forEvaluation. Due consideration will also be given to beneficiaries and other stakeholders on confidentialityof information and privacy during consultations and personal interviews.7. DOCUMENTATIONThe following documents will be made available to the Consultants to facilitate the evaluation:(i)(ii)(iii)(iv)(v)(vi)(vii)(viii)(ix)(x)(xi)Evaluation TOR;Country Programme Document (CPD) and Country Programme Action Plan (CPAP);Country Office Annual Reports (COARs);2011 Moldova UNDAF Evaluation Report;Relevant donor reports;Training manuals;Training and workshop reports;Evaluation reports of earlier phases;Annual work plans;Budget sheets;UNFPA Evaluation Guide.8. EVALUATION MANAGEMENT - ROLES AND RESPONSIBILITIESThe Evaluation team will work directly and closely with the UNFPA Coordination and Support Team(CST) which will be set up by UNFPA Chisinau CO to provide a) overall guidance and monitoring ofquality Assurance at the UNFPA National Coordinator level and b) all the support required (logistics,meetings with stakeholders, dissemination, and debriefing meetings at the UNFPA Admin/FinanceOfficer level.The Coordination and Support Team (CST) will meet with the evaluation consultants at agreed schedules,discuss progress and address any challenges faced. With overall guidance and support from the CountryDirector the CST will be responsible in particular for the following:a. Provide overall technical guidance and quality assurance on every step of the evaluation process;8

. Review and endorse the terms of reference of the evaluation;c. Recommend the TOR to the UNFPA Country Director and submit the same to EEACRO andDOS for review and approval in accordance with UNFPA evaluation guidelines;d. Select and secure appropriate endorsement of consultants;e. Review and approve inception:f. Coordinate and set up meeting schedules with beneficiaries and stakeholders in consultation withGovernment;g. Facilitate the evaluation consultants’ access to all background documents;h. Make travel arrangements and in-country consultations with programme beneficiaries andpartners;i. Allocate office space to the Consultant/evaluator for work and review;j. Coordinate UNFPA internal review processes; andk. Coordinate with UNFPA management approval of all evaluation deliverables.The UNFPA Programme Coordinator shall be responsible for overall quality assurance of the evaluationin accordance with UNFPA and DOS Evaluation guidelines.The final report will be submitted to the UNFPA Country Director for review and approval. UNFPAmanagement will then prepare management response to the evaluation findings and recommendations.The evaluation consultants will interact primarily with a Government Counterpart Evaluation Group(GCEG) comprising of the following officials:1. Mr. Victor Lutenco, Adviser on Social Affairs to the Prime Minister of Moldova2. Mr. Sergiu Sainciuc, Deputy Minister of Labour, Social Protection and Family (MLSPF)3. Ms. Rodica Scutelnic, Head of Department, MoH4. Nadejda Velisco, Head of Department, MoE5. Ms. Lilia Pascal, Head of Department, MLSPFThe GCEG individually and as Group will provide the evaluation consultants their perspective andexpectations broadly and sectorally at the policy, programme implementation, performance, institutionalcapacity building and impact level. The Group may also recommend to the evaluation consultants othercounterparts to meet in both government and civil society9

9. DELIVERABLESA. Inception ReportWithin five (5) days of award of contract, the Team leader, based on consultations with the otherconsultant shall submit an electronic copy of a draft inception report to UNFPA CO. The inception reportprovides an opportunity for UNFPA CO and the Consultants to ensure that their interpretations of theTOR are mutually consistent. The CO will coordinate the internal review and approval of the inceptionreport before submitting the report to the UNFPA Country Director for review and feedback. Theinception report shall:• Explain the evaluators’ understanding of what is being evaluated and why;• Describe the strategy for ensuring the evaluations’ utility and applicability to the needs of UNFPAand those of key stakeholders;*• Review and strengthen the evaluation methodology, describing the plans to engage and involvestakeholders in the design (e.g., questions, objectives, methods, data-collection instruments), datacollection, data analysis, and development of recommendations;*• Explain how the evaluation questions will be addressed with respect to all evaluative criteriaindicated above by way of proposed methods, evaluation designs, sampling plans, proposed sourcesof data, and data-collection procedures;*• Note: Consultants/Evaluators are encouraged to suggest refinements to the TOR and to proposecreative or cost- or time-saving approaches to the evaluation and explain their anticipated value.• For each of the evaluative criteria, describe the measurable performance indicators or standards ofperformance that will be used to assess progress towards the attainment of results, includingoutcomes;*• Discuss (a) the limitations of the proposed methods and approaches, including sampling, with respectto the ability of the evaluation team to attribute results observed to UNFPA’s efforts especially in theabsence of a valid counterfactual and (b) what will be done to minimize the possible biases andeffects of these limitations;*• Explain the Consultants’/Evaluators’ procedures for ensuring quality control for all deliverables;• Explain the Consultants’/Evaluators’ procedures to ensure informed consent among all people to beinterviewed or surveyed and confidentiality and privacy during and after discussion of sensitive issueswith beneficiaries or members of the public;*• Indicate familiarity with and agreement to adhere to (a) the requirements of the Standards forEvaluation in the UN System, especially standards 4.1 through 4.18 and (b) UNFPA’s EvaluationQuality Standards, which will be provided to the evaluator; and,• Provide a proposed schedule of tasks, activities, evaluation methodologies and deliverables consistentwith this TOR.10

The Team Leader, assisted by the other consultant shall make oral presentation/briefing of the inceptionreport to UNFPA CO and its stakeholders. UNFPA Programme Coordinator shall obtain writtencomments on the inception report from the Government Counterpart Evaluation Group to theConsultants/Evaluators within 2 days of the report’s submission or completion of the oral presentation,whichever comes later. UNFPA CO reserves the right to modify the TOR in response to the inceptionreport.Note: Items marked with an asterisk should also be discussed in the evaluation report.B. Draft Evaluation ReportThe Evaluator/ Team Leader shall submit an electronic copy of a draft evaluation report to UNFPA COno later than 21 days after completing CP evaluation. The draft report should be thoroughly copy editedto ensure that comments from the UNFPA and other stakeholders on content, presentation, language, andstructure can be reduced to a minimum.The Team Leader shall make a debrief presentation to the UNFPA CO on submission of the draftevaluation report.After UNFPA’s and stakeholders’ review of the draft report, the UNFPA Programme Coordinator shallcoordinate written comments on the draft report from UNFPA CO and in-country partners and shallsubmit these to the Team Leader. Based on these comments, the Team leader, with the help from theother consultant shall correct all factual errors and inaccuracies and make changes related to the report’sstructure, consistency, analytical rigor, validity of evidence, and requirements in the TOR. The evaluatorswill not be required to make changes to conclusions and recommendations unless they are regarded asqualitative improvements. After making the necessary changes, the Team leader will submit a reviseddraft evaluation report, which may lead to further comments from UNFPA. After the second round ofreview and, if necessary, further revision to the draft evaluation report, the Team leader can then submitthe final report pending UNFPA’s approval.C. Final ReportThe recommended structure of the final report is provided below and the Team Leader as well as otherConsultants/Evaluators should follow this as closely as possible. The report must contain a self-containedexecutive summary that provides a clear, concise presentation of the evaluation’s main conclusions andkey recommendations and reviews salient issues identified in the evaluation. All deliverables must be inEnglish.Structure of ReportThe final evaluation report should contain the following:• Title Page• List of Acronyms and abbreviations• Table of Contents, including list of annexes11

• Executive Summary. The executive summary should be brief (1-2 pages) and contain key informationneeded by decision makers. It should contain the following:o Brief description of the CPAPo Evaluation purpose, questions and scope of evaluationo Key findingso Conclusionso Key recommendations• Introduction: background and context of the project and project evaluation• Description of the project – its logic theory and change theory, results framework and external factorslikely to affect success;• Purpose of the Evaluation• Key questions and scope of the evaluation with information on limitations and de-limitations• Approach and methodology of the evaluation• Findings• Summary and explanation of findings and interpretations• Conclusions• Recommendations• Lessons Learned, good practices, generalizations and alternatives• AnnexesAt a minimum, the final report shall contain the following annexes:• List of persons interviewed (if confidentiality permits) and sites visited;• Data-collection instruments (copies of surveys, questionnaires, etc.);• A bibliography or list of references; and,• The TOR for the evaluation.This evaluation report template is intended to serve as a guide for preparing meaningful, useful andcredible evaluation reports that meet quality standards. It does not prescribe a definitive section-bysectionformat that all evaluation reports should follow. Rather, it suggests the content that should beincluded in a quality evaluation report. The descriptions that follow are derived from the UNEG‘Standards for Evaluation in the UN System’ and ‘Ethical Standards for Evaluations’.10. CONSULTANCY/EVALUATION SCHEDULES12

The overall consultancy will tentatively begin on 27 June 2011 and finalize in 15 working days. It willinclude one 7-10 days visit to Moldova for the international consultant, to undertake consultations withUNFPA CO and partners. The other two and three days of remote work will be dedicated to the deskreview and wrap up of the mission/ finalising of the report, respectively. Attached as Annex 5 is thedetailed evaluation schedule.The Team leader/ International Consultant is expected to submit a draft evaluation report, drafted basedon inputs received from the other consultant, with a debriefing with the UNFPA CO no later than 15 July2011.The Consultants will be remunerated according to the following schedule: (a) 30 percent of payment uponcompletion of a satisfactory inception report; (b) 20 percent upon successful completion of field work;and, (c) 50 percent upon submission of a satisfactory final report.11. CONSULTANT/EVALUATOR QUALIFICATION AND EXPERIENCEThis evaluation requires a team of two consultants: one International consultant acting also as TeamLeader to evaluate RH and P&D components and one National consultant to evaluate Gender component(as a result to cover all three CP components). The team should have the following profile(s).International Consultant to cover P&D and RH components is required:Academic Qualifications and experiences:• Advanced university degree (Master's or equivalent) in Health Economics, Social Policy, Publichealth, Gender, Demography, Development or transition Studies, or relevant Social Sciences withat least 8-10 years of relevant professional experience, including previous substantial involvementin evaluations and/or reviews;• Specialized experience and/or methodological/technical knowledge, including data collection andanalytical skills, particularly in one of the following areas: human rights-based approaches toprogramming; gender considerations; Results Based Management (RBM) principles; logicmodelling/logical framework analysis; quantitative and qualitative data collection and analysis;participatory approaches; Sector Wide Approaches (SWA).• Knowledge of the UN system• Excellent knowledge of the UN system and UN common country programming processes,UNDAF and UNFPA programme activities; Knowledge of UN reforms and Delivering as One isdesirable;• Past experience as a team-leader in a related assignment (s);• Ability to work in partnership with various stakeholders: partners, governments, beneficiaries onsensitive issues like reproductive health;• Excellent report writing, communication, interviewing and computer skills;• Awareness and sensitivity in working with people of various cultural and social backgrounds.The National Consultant to evaluate the Gender component of the CPAP:13

• Advanced university degree in Social Policy, Sociology, Gender Studies, Development Studies,or other relevant Social Sciences with at least 8 years of relevant professional experience,including previous involvement in evaluations and/or reviews. PhD is an advantage but notrequired.• Substantial experience in development related work, preferably in workers education and familiarwith gender equality issues and concerns at national, regional and international levels, extensiveexperience in conducting evaluations and proven record in delivering results within a shorttimeframe, previous experience with other UN agencies;• Experience in the evaluation of UN supported programmes will be an added advantage;• Proven experience in policy development and analysis around gender issues and povertyreduction strategies highly desirable;• Experience and skills in using evidence-based, knowledge base creation and ability to developsystems for improved performance;• The Evaluation Report will be prepared in English. A good command and knowledge of theEnglish language is essential. Knowledge of Romanian or Russian will be an asset.• Excellent report writing, communication, interviewing and computer skills;• An understanding of and ability to abide by the values of the United Nations;• Awareness and sensitivity in working with people of various cultural and social backgrounds.14