Terms of Reference - The Regional Environmental Center for ...

Terms of Reference for Project EvaluationPromoting regional cooperation in South Eastern Europevia networking within the authorities responsible for theenvironment and justice sectors (Themis Network)1. Project descriptionProject locationProject nameProject reference number 8284-00/2011Project budget EUR 700,000Donor(s)/funding sourcesAlbania, Bosnia and Herzegovina, FYR Macedonia,Montenegro, Serbia, Kosovo*Promoting regional cooperation in South EasternEurope via networking within the authoritiesresponsible for the environment and justice sectors(Themis Network)Austrian Development AgencyProject duration January 1, 2012 to March 31, 2014lmplementing agency andpartners1. BackgroundThe Regional Environmental Center for Central andEastern Europe (REC)The project goals are to strengthen regional cooperation in the SEE region and increase theadministrative capacities of the relevant national authorities, namely in the environment andjustice sectors, in the fields of natural resources management and environmental crimes. It isexpected that these efforts will result in a better preparation for EU accession anddeveloped environmental law enforcement mechanisms.The expected results are increased capacities in the relevant national authorities, includingthe justice sector, and cooperation activities, by means of well-trained key personnel,enhanced internal dialogue and regional communication tools.The project activities are grouped into three clusters:1) The management of the Themis Secretariat and the organisation of ExecutiveCommittee meetings.2) The protection of natural resources and combating illegal logging by developing aRegional Forest Law Enforcement and Trade Action Plan, capacity-building activitiesand technical assistance on law and policy development3) Combating environmental crimes through capacity building and cooperationactions with international partners.* This designation is without prejudice to positions on status, and is in line with UNSCR 1244 and the ICJ Opinion onthe Kosovo declaration of independence.

This project targets 120 people trained and involved in the activities directly, with a spillovereffect to another 430 as a minimum. The target groups are policy makers and legislators fromministries of justice and environment, judges and prosecutors, and practitioners from theenforcement agencies attached to the ministries of environment, forestry, and the interior.The Regional Environmental Center (REC) is implementing the project, as the secretariat forthe Themis cooperation platform, with the help of its country office network in thebeneficiary region.2. Purpose of the evaluationThe evaluation is intended to assess the relevance, performance, managementarrangements and successes and failures of the project, based on the benchmarks andindicators identified during the planning stage.The evaluation will, in particular:• seek to enhance the project impact by examining lessons learned for a possible newphase of the project;• provide accountability to the donor by determining whether the project objectiveswere met and resources were wisely utilised;• identify areas of improvement in the project;• clarify and tackle problems; and• obtain participatory feedback from the major stakeholders involved in the project.The evaluation will also:• provide suggestions on setting different priorities and goals;• provide suggestions on new strategic directions for the future; and• provide recommendations on the methodology, planning and implementation ofsimilar projects in the future, always keeping in mind that the final objective would beto contribute to the development of environmental protection in the beneficiaryregion.The main users of the evaluation will be the project management team, the donor and thebeneficiary countries.3. ObjectivesThe evaluation is commissioned by the REC, as implementing partner, following the rules ofthe donor (ADA) for accountability and quality assurance. The exercise is carried out in thefinal months of project implementation to assess the adequacy of the proposed instrumentsand the results achieved.The evaluation aims to review the results (outputs, outcomes, impacts) and assess theeffectiveness, efficiency and relevance of interventions with a view to providing conclusionsand recommendations.4. Subject and focus (scope)Unit of analysis and time period:The evaluation covers activities performed under the project “Promoting regionalcooperation in South Eastern Europe via networking within the authorities responsible for theenvironment and justice sectors (Themis Network)”, Contract n° 8284-00/2011 for the period2012-2013.Financed by Austrian Development CooperationPage 2 of 8

Interventions covered and target groups:The evaluation will analyse the activities carried out under the three clusters of the projectand the accomplishment of the expected results, in particular for the capacity-buildingactivities and cooperation actions, as well as the overall running of the activities (projectmanagement, communication). It will also cover the project benefits for the target groupstogether with the eventual weaknesses in their targeting and needs assessment.Geographical scope:The evaluation will be executed at regional level, as the project followed a regionalapproach. However, national-level benefits and impacts are to be identified for thebeneficiary countries whenever possible.Funds for interventions:The evaluation will provide an analysis of the overall adequacy of funds and of whether theresources were wisely utilised, without prejudice to the financial audit of expenditures that isregularly carried out for the financial reporting.Cross-cutting issues:The addressing of gender equality and anti-corruption transparency measures will be takeninto consideration in the analysis of the results. In particular, it will be checked whether thegender considerations integrated in the project document (e.g. gender disaggregation inthe purpose indicators) have been addressed.Logframe analysis:The intervention logic will be analysed.Other parameters:The participation of the beneficiary countries and country ownership will be assessed,together with coordination issues and communication (internal/external) by the projectmanagement team.The evaluation will rely on the five OECD DAC evaluation criteria (relevance, efficiency,effectiveness, impact and sustainability) and the criteria of partnerships and cooperation.The evaluators shall follow all the guidelines provided in the publication “Guidelines forProject and Programme Evaluations”, available on the Austrian Development Cooperationwebsite.5. Main evaluation questions1) Relevance• How relevant is the project and its activities to the development and transformationof the beneficiary region?• How relevant is the project in the process of EU law implementation andenforcement in the field of environment (natural resources, forestry, environmentalcrimes prevention and prosecution)?• How relevant is the project to the beneficiary countries’ needs and priorities/strategic goals in the field of environmental law implementation and the EUaccession process?• How important is the significance of the interventions regarding the target groups’needs (including women) and the strengthening of national authorities for bettercompliance?• How relevant are the actions to the intervention logic set at the project designphase?Financed by Austrian Development CooperationPage 3 of 8

• To what extent does the project comply with ADA’s strategic goals for thebeneficiary region?2) Efficiency• Were the resources and inputs (man-days and budget) converted to outputs in atimely and effective manner?• Were the capacities used in a reasonable way?• What is the cost/benefit ratio? Were the initial resources proportional to the outcomes(i.e. was the project efficient overall)?• Was the monitoring of the implementation adequate from the project managementside?• Was cooperation from the side of the financial support recipients adequate(beneficiary countries/institutions)?• What was the effectiveness or impact of the project activities compared to theobjectives planned?3) Effectiveness• Were the planned objectives and outcomes contained in the project documentachieved? Were these objectives realistic?• Have the project activities contributed to sustainable capacity building in thebeneficiary countries?• What factors were crucial for the achievement of/failure to achieve the projectobjectives?• What other factors can be observed regarding the achievement of the projectobjectives?• Was the outreach of activities towards beneficiaries adequate (trained personnel)?• Were materials developed and prepared? What was the quality?• Were the planned cooperation actions with partners carried out? Were newpartnerships formed or existing ones strengthened?• Were the planned guidelines/publications produced? Were they adequate to thecountries’ needs?• Were the horizontal cross-cutting issues addressed?• If the planned outcomes were not achieved, what measures can be suggested tothe project management?• Were internal and external communication tools properly used to reach a widetarget audience?4) Impact• To what extent were the overall objective and project goals realistic?• Did the interventions contribute to reaching a higher level than the objectives set atthe project design phase?• What is the impact or effect of the interventions in proportion to the overall situationof the target group, also including gender issues?• Has the project contributed, or is it likely to contribute, to long-term social, economicand technical changes for the countries, individuals and institutions benefiting fromthe project?• What is the status of project ownership in the beneficiary countries? Did it increaseduring the project?5) Sustainability• Are the positive effects or impacts sustainable?• How is the sustainability or permanence of the intervention and its effects to beassessed?Financed by Austrian Development CooperationPage 4 of 8

• To what extent are the project results likely to continue after the project?• What risks/potentials can be detected as regards the sustainability of project results?• Will the beneficiary countries contribute in some form to the sustainability of theproject processes and results?• To what extent was the intervention exemplary? To what extent did it create lastingstructures and mechanisms? 6) Partnerships and cooperation• To what extent have sustainable partnerships been sought and established andsynergies created in the delivery of assistance?• How have regional cooperation and partnerships between the countries developedduring the project implementation and beyond?6. Evaluation approach and methodsThe evaluation will be carried out by external evaluator(s) in line with the OECD DACEvaluation Quality Standards and following the ADC “Project and Programme Evaluation”guidelines and international standards.The evaluator(s) are expected to elaborate an inception report and a final evaluation reportresulting from desk research based on the available project documents and from bilateralinterviews with the project beneficiaries.The following evaluation phases are envisaged:1) The evaluator(s) will be requested to commence the assessment with deskresearch, by screening the project documents and the implementation files(reports, monitoring indicator tables, evaluations of the training activities, contactlists for the beneficiary countries). A meeting might be organised to debrief theevaluators.2) Based on the information collected, a draft inception report will be prepared,which will be discussed with the project team with a view to receivingcomments/clarification requests. For the report, preliminary data will be analysedalso taking into account horizontal issues (e.g. gender, anti-corruption,transparency), and a final methodology will be agreed upon. The evaluators willproduce the inception report in compliance with the format available as anannex in the ADC Guidelines for Project and Programme Evaluation.3) The evaluator(s) will then conduct definitive information and data gathering,through whatever means approved in the inception report, and will take care toinclude the opinions of the beneficiary countries’ representatives, project team,partners, and any other stakeholders involved in their assessment. The datacollection will be followed by final data analysis and the drafting of theevaluation report.4) The draft final report will be sent to the project team, project beneficiaries andADA for quality control and comments. Once the comments have been taken onboard the report will be finalised and made available to the project stakeholders.The total number of days dedicated to the task should be 15 working days, including on-thespotvisits, if applicable, with a maximum duration of four days.Financed by Austrian Development CooperationPage 5 of 8

7. TimetableActivities Timeline: Deadline Chargeable workdays0. Tender acceptance October 10, 2013 NA1. Desk research, screening of projectdocumentsOctober 20, 2013Financed by Austrian Development Cooperation2 daysInception report October 25, 2013 3 days2. Gathering and analysinginformation in preparation for thefinal report3. Draft final report completed andsent for comments and qualitycontrol4. Final report, with commentsaddressed, and publicdissemination8. Evaluator’s team and qualificationsNovember 10, 2013November 15, 2013December 15, 20135 days2 days3 daysMain qualifications required from evaluator(s):• Profound knowledge and experience in environmental governance, lawenforcement and EU environmental law.• Experience with the EU accession and law harmonisation process.• Understanding of public sector challenges and opportunities, with a view toinstitution development.• Knowledge of the specificities of the beneficiary countries and the dynamics ofpolitical, economic, social transition, in particular linked to the environmentalsector.• Proven experience in project management and evaluation. Internationalexperience within the United Nations system/EU policies will be considered anasset.• Previous experience in South Eastern Europe and of capacity-building projects.• Cross-sector and gender expertise.• Excellent drafting skills.• Fluency in English required. Knowledge of the national language(s) of thebeneficiary countries will be considered an asset.Candidate(s) must submit to the REC an application package containing a technical offerand a financial offer in two separate sealed envelopes, together with an electronic copy ofthe documents on a CD. The technical offer must include a cover letter of a maximum oftwo pages presenting the evaluator/team and illustrating the evaluator’s/team members’capacities, relevant experiences and qualifications for the envisaged tasks. For teamcandidates, it must be clearly visible from the offer which of the proposed evaluators will actas team leader. The cover letter should be accompanied by supporting documents (CVs,references, relevant certificates). The financial offer must be submitted in a separateenvelope as the two will be assessed independently.In order for offers to be assessed not only according to the aspect of costs, the technicaloffers will be assessed first by the project team at the REC, based on a pre-definedevaluation grid. The assessment team will comprise the REC staff members responsible forPage 6 of 8

project implementation (maximum four people), the project director and one representativefrom the REC’s Finance Department. The best value for money offer will be selected.The deadline for the submission of applications is 14:00 CET on August 30, 2013.Offers must be sent by post to:ATTN: Aniko Nemeth/ Bruno MesquitaRegional Environmental CenterAdy Endre ut 9-112000 Szentendre, Hungary9. ReportsThe output of this task will be an evaluation report in English. The maximum length of thereport is 40 pages with annexes. As preparation for the evaluation task and final report, it isalso expected that the evaluator(s) will produce an inception report following the formatdefined in the ADC Guidelines for Project and Programme Evaluation.The structure and content of the final report should meet the requirements of the ADC“Project and Programme Evaluation” guidelines and other international standards. The finaldraft evaluation report and final report must be structured according to the OECD/DACcriteria and the evaluation questions.The evaluation report must:• contain an executive summary (mandatory);• be analytical in nature (both quantitative and qualitative);• be structured around issues and related findings/lessons learnt;• include conclusions; and• include recommendationsThe evaluation report will be assessed against the evaluation quality criteria of theOECD/DAC. In addition, the final report should:• comply with these terms of reference;• include a separate analysis of the cross-cutting issues;• include a description and assessment of the intervention logic;• base any conclusions and recommendations on clearly stated findings;• include a clear methodology section (previously defined and agreed in theinception report) explaining how information was gathered, processed andanalysed;• be realistic about the lessons learnt and other recommendations, which should beclearly addressed to concrete stakeholders, partners, etc.;• contain a list of consulted stakeholders; and• contain a bibliography detailing all documents used, but highlighting the mostimportant. Financed by Austrian Development CooperationPage 7 of 8

10. Coordination and responsibilityThe evaluator(s) shall be duly responsible for fulfilling the tasks described. The REC will providesupervision and logistical support for the desk research, by making the relevant projectdocuments available, and for setting up interviews on the spot, if applicable.11. AnnexesProgramme details.Please consult the project website http://themis.rec.org/. Most documentation aboutproject activities is posted on this site. More detailed information, or information not disclosedto the public, will be made available to the selected team of evaluators.Financed by Austrian Development CooperationPage 8 of 8