accession program for agriculture and rural development in ... - MADR

Terms of Reference – SAPARD ProgramUPDATING OF MID - TERM EVALUATION OF THE SPECIAL PRE-ACCESSION PROGRAM FOR AGRICULTURE AND RURALDEVELOPMENT IN ROMANIAFOR THE PERIOD 2000 - 2005- February 2005 -

Index1. GENERAL INFORMATION1.1 Beneficiary...............................................................................................................................................31.2 Relevant data about beneficiary ...........................................................................................................31.3 Relevant data about beneficiary country.................................................................................................41.4 Business framework in the relevant sector .............................................................................................51.4.1 SAPARD Program in Romania............................................................................................51.4.2 Context of the Mid-Term Evaluation Updating of theSAPARD Program in Romania............................................................................................81.5 Correlated Programs and other donors activities .............................................................................102. CONTRACT OBJECTIVES2.1 General Objectives..................................................................................................................... 112.2 Specific Objectives..................................................................................................................... 113. CONSIDERATIONS AND RISKS3.1 Considerations emphasizing the necessity of the project ......................................................... 133.2 Risks .......................................................................................................................................... 144. FIELDS OF ACTIVITY4.1 Geographic scope and time frame ............................................................................................. 144.2 Regulatory Scope ....................................................................................................................... 144.3 General and Specific obligatory Activities .................................................................................. 144.3.1 Mid-Term Evaluation Updating Steps............................................................................ 154.3.2 Obligatory elements of the Mid-Term Evaluation Updating........................................... 164.3.2.1 Analysis of the ex-ante Evaluation and theMid-Term Evaluation 2000-2003 results ....................................................... 164.3.2.2 The validity of SWOT analysis carried out during theProgram ex-ante Evaluation Program ........................................................... 174.3.2.3 Evaluation of adequacy and consistency of theDevelopment Strategy in NPARD.................................................................. 184.3.2.4 The Quantification of Goals – Outputs, Results and Impact ........................ 194.3.2.5 Evaluation of the effectiveness and efficiency of Program Implementation –Evaluation of policy and distribution of resources ......................................... 20Efficiency – inputs, compared with outputs and results................................. 20Effectiveness – outputs and results compared with specificand operational objectives ............................................................................. 21Impact - global objectives .............................................................................. 214.3.2.6 Quality of Program implementation and organization of Monitoring ............. 224.3.2.7 Institutional impact ........................................................................................ 234.3.2.8 Conclusions and recommendations .............................................................. 234.3.2.9 Structure of Mid-Term Evaluation Updating Report ...................................... 254.3.2.10 Bibliography................................................................................................ 284.4 Project management, contractor’s tasks and responsibilities .................................................... 31Obligations of the Contracting Authority in Romania.............................................................. 31Obligations of the Contractor.................................................................................................. 32Awarding criteria ..................................................................................................................... 334.5 Independence of the Evaluator................................................................................................... 36TERMS OF REFERENCE - Updating of Mid-Term Evaluation of the SAPARD Program 1

5. LOGISTICS AND PROGRAMMING5.1 Place where the project is executed.......................................................................................... 375.2 Period of the Project .................................................................................................................. 376. REQUIREMENTS6.1 General Requirements ............................................................................................................... 386.2 Contractor Requirements ........................................................................................................... 386.3 Key Experts Requirements......................................................................................................... 396.3.1 Profile of International Experts.......................................................................... 396.3.2 Profile of Romanian Experts............................................................................. 417. REPORTS7.1 Inception Report........................................................................................................................ 457.2 Activity Reports ......................................................................................................................... 457.3 Mid-Term Evaluation Updating Report...................................................................................... 458. MONITORING AND EVALUATION8.1 Evaluation criteria ....................................................................................................................... 468.2 Special requirements.................................................................................................................. 48TERMS OF REFERENCE - Updating of Mid-Term Evaluation of the SAPARD Program 2

In accordance with relevant EC regulations, at the bases of SAPARDProgram implementation in Romania, there is the National Plan forAgriculture and Rural Development (NPARD) elaborated by the Ministry ofAgriculture, Forests and Rural Development.The NPARD, preceded by a 2.5 million € test program financed through thePhare Project RO/98/IB/SPP/02, presents the strategy to be followed in theperiod 2000-2006 with the aim of improving the access to markets and thecompetitiveness of agricultural products, ameliorating the infrastructures,and developing rural economy as well as human resources.Giving the fact that needs for investments in agriculture are considerablethe Romanian Government has been developing complementary actions tothe SAPARD Program. Among them we could mention: the setting-up of apreferential credit system for agricultural producers; the coupon system; thesupport to young farmers and to entitled persons; the support to poorcommunities and productive groups in rural space by the Romanian Fundfor Social Development (RFSD); the special programme for less-favouredareas “Assistance for agricultural activities in rural area”, granting financialnon-reimbursable aid for the purchase of machines and equipment foragricultural productive activities or specific services; the negotiations withinternational donors to get financial support for agriculture and ruraldevelopment.The National Plan for Agriculture and Rural Development is a documentwhich may be improved and which will be updated on the basis of thenational strategy on economic and social cohesion; of the EuropeanCommission policies on agriculture and rural development funding; andalso of the findings of monitoring activities carried out in the framework ofthe present contract.1.4 Business framework in the relevant sector:1.4.1 SAPARD Program in RomaniaThe Special Pre-Accession Program for Agriculture and Rural Development– SAPARD – represents the financial contribution of the Community,established by the Council Regulation (EC) 1268/1999 of 21 June 1999concerning the support of the Community for the pre-accession measuresin the field of agriculture in the candidate countries from East and CentralEurope, during the pre-accession period. The general objectives of thisTERMS OF REFERENCE - Updating of Mid-Term Evaluation of the SAPARD Program 5

Program are: 1) to contribute to the implementation of the acquiscommunautaire concerning the CAP and related policies, and 2) to solvepriorities and specific problems for the sustainable adaptation of theagricultural sector and rural areas of candidate Countries.On 21 st of June 2000, the European Commission decided on an annualindicative allocation for each candidate country within a total global amountof about 520 million Euro/year, over the period 2000-2006.For Romania, the annual allocation amounts to about 153 million Eurowhich places our country the second on top, after Poland, amongbeneficiary countries.The National Program for Agriculture and Rural Development (NPARD)presented by the Romanian Government was examined by the STARCommittee of the European Union and approved officially by the EuropeanCommission by the Decision no. EC (2000) 3742 of 12 December 2000.NPARD is the programming document which covers the whole rural areasand ensures the implementation of the first operational Program - theSAPARD Program. In the following years it has been modified four times,respectively by European Commission Decision No. H/2002/1936 on 12 ofJuly 2002, European Commission Decision on 27th of May 2003, EuropeanCommission Decision on 1st of August 2003 and European CommissionDecision on 17th September 2004.The strategic objectives of the Program in Romania aim at:• Priority 1: Improving the access to markets and of thecompetitiveness of agricultural processed products;• Priority 2: Improving infrastructures for rural development andagriculture;• Priority 3: Development of rural economy;• Priority 4: Development of human resources.For the technical and financial implementation of the SAPARD Program,the Romanian Government adopted on 14 th of September 2000 theGovernment Urgency Ordinance no.142/2000, approved by Law no.309/2001 which establishes the setting up of the SAPARD Agency asinstitution subordinated to the Ministry of Agriculture, Food and Forests,organized at central and regional level (8 regional offices corresponding tothe 8 development regions of the country).TERMS OF REFERENCE - Updating of Mid-Term Evaluation of the SAPARD Program 6

The Multi-Annual Financing Agreement, which defines the rules of theSAPARD Program implementation, was signed by the Commission ofEuropean Communities and the Government of Romania on 2 nd ofFebruary 2001 and ratified by Law no.316/2001.The Annual Financing Agreements for 2000, 2001, 2002 and 2003 whichestablish the financial commitment of the Community for Romania for eachof the above mentioned years as well as the period of validity were signedby the Commission of European Communities and the Government ofRomania on 27 th of February 2001, 30 th of January 2002, 1 st of April 2003and 31 st of July 2003 respectively and subsequently ratified by Law no.317/2001, Law no. 416/2002, Government Urgency Ordinance n. 26/2003and Law no. 496/2003 respectively.On 11 th of July 2002, the Commission approved Romania’s proposals toamend the NPARD. The need to amend the NPARD resulted, mainly, inthe process of finalizing the specific procedures and operational structuresin view of the SAPARD Agency accreditation and conferral of financialassistance management to Romania.By the European Commission Decision (EC) 638/2002 of 31 st of July 2002,the SAPARD Agency was accredited and was conferred the decentralizedmanagement of the financial assistance provided by the SAPARD Programfor the implementation of three measures:-Measure 1.1 - “Improvement of processing and marketing of theagricultural and fishery products”,-Measure 2.1 - “Development and improvement of rural infrastructure”,-Measure 4.2 - “Technical assistance”.On 5 th of December 2003, the European Commission conferred to theRomanian authorities the management for other three measures of theSAPARD Program:-Measure 3.1 - “Investments in agricultural holdings”,-Measure 3.4 - “Development and diversification of economicactivities, multiple activities, alternative incomes”,-Measure 4.1 - “Improvement of the vocational training”.The other measures contained in the National Program for Agriculture andRural Development (NPARD) will be accredited in 2005, namely:TERMS OF REFERENCE - Updating of Mid-Term Evaluation of the SAPARD Program 7

-Measure 1.2 - “Improving the structures for quality, veterinary, andplant health controls, foodstuff quality and consumerprotection”,-Measure 2.2 - “Management of water resources”,-Measure 3.2 - “Setting up producers groups”,-Measure 3.3 - “Agricultural production methods designed to protectthe environment and maintain the countryside”,-Measure 3.5 - “Forestry”.1.4.2 Context of the Mid-Term Evaluation Updating of the SAPARDProgram in RomaniaThe general framework which defines the type of evaluations to beachieved in the different stages of the Program cycle – ex-ante evaluation,mid-term evaluation and ex-post evaluation – is ensured by the RegulationEC 1268/1999, Article 5 (1) and Regulation EC 1260/1999, Chapter IV.The actual modalities to achieve the mid-term evaluation, and its updating,are provided by the Multi-Annual Financing Agreement, Section B, Article 5(5), Article 9 and Article 10, as follows:“The Managing Authority shall organize, in cooperation with theCommission and Romania, the mentioned mid-term evaluation.Mid-term and ex-post evaluation shall respond to common evaluationquestions defined by the Commission in consultation with Romania andshall, as general rule, be accompanied by achievement related criteria andindicators. In addition, evaluations may be required to answer specificquestions to the objectives of the Program.Romania shall ensure that a mid-term evaluation examines the initialresults of the Program, their consistency with the ex-ante appraisal, therelevance of the targets and to what extent they have been attained. It shallalso assess the quality of monitoring and implementation.The mid-term evaluation shall be carried under the responsibility of theManaging Authority in cooperation with the Commission and Romania.It shall be submitted to the Monitoring Committee and shall be sent to theCommission three years after the date of approval of the Program but by31 December 2003 at the latest.TERMS OF REFERENCE - Updating of Mid-Term Evaluation of the SAPARD Program 8

The Commission shall consider the implications of the evaluation with aview to reviewing the Program.The mid-term evaluation shall, where appropriate, be updated andsubmitted to the Commission by 31 December 2005 at the latest.The Managing Authority shall inform the Commission about the follow-up tothe recommendations in the evaluation report, including any possibleupdates”.The updating of mid-term evaluation was proposed to the 6 th MonitoringCommittee within the Action Plan for technical assistance measure, inconsideration of the implementation progress of the measures accredited in2002 and 2004 during the period 2003-2005, not covered by the mid-termevaluation for 2000-2003, as well as of the fact that some measures’implementation had not yet started. The 7 th Monitoring Committee, held inOctober 2004, approved the updating of mid-term evaluation and instructedthe Monitoring Committee Secretariat to organize the relevantprocurement.In completion to the mentioned documents, the European Commissionissued the “Guidelines for Evaluation of rural development Programs 2000-2006 supported from the European Agricultural Guidance and GuaranteeFund” (Doc. DG AGRI VI/8865/1999) and the “Guidelines for the mid-termevaluation of rural development Programs 2000-2006 supported from theEuropean Agricultural Guidance and Guarantee Fund” (Doc. STAR VI/43517/2002) which include:• Definition, scope, concept and types of evaluation• Setting up, organisation and implementation of evaluation strategy• Methodology of evaluation• Carrying out of the actual mid-term evaluation• Common evaluation questions with criteria and indicators andProgram specific questions with criteria and indicators• Evaluation Report structure• Possible updating of the mid-term evaluationThe above mentioned papers as well as the “Guidelines for the Evaluationof Rural Development Programmes Supported by SAPARD” (Doc. DG-TERMS OF REFERENCE - Updating of Mid-Term Evaluation of the SAPARD Program 9

Agri, 4.2001) and the “Guidelines for the mid-term evaluation of ruraldevelopment Programs funded by SAPARD 2000-2006” (Doc. DG-AGRI09/2002) should be considered as complementary to these ToR.1.5 Correlated Programs and other donors activities:1. The project “Rural Development in Romania” (PHARE RO 9505-04-03-005), aiming to the definition of a complex rural development strategyand to the creation of an intervention framework at institutional level for theimplementation of rural development policies. Project results aimed at theelaboration of a diagnosis study of Romanian rural space (with reference to2,685 municipalities), through which 45 indicators have been analysed. Onthe basis of this study, areas characterised by different developmentpotentials have been determined. A Green Book has also been elaboratedconcerning Rural Development in Romania and an Action Plan for RuralDevelopment has been adopted in the shape of an inter-ministerialdocument.2. The project «Institutional Building for Ministry of Agriculture and Foodfor SAPARD Program implementation» (PHARE RO 98/SPP/02), having asmain objective the capacity strengthening of Ministry of Agriculture andFood in order to elaborate a structural policy and rural developmentstrategy for the implementation of the SAPARD Program measures.Theprojects result consisted in the drawing up of the National Plan forAgriculture and Rural Development (NPARD), insuring the legislative andinstitutional framework for the SAPARD Agency setting up, its staff trainingat central and local level for the technical implementation of the Program,and the preparation of the Special Preparatory Program (SPP)implementation.3. Special Preparatory Program SPP (PHARE RO 9807-01-03-05),which has represented a preparatory exercise for the SAPARD Programimplementation in Romania.4. Project EUROPEAID/114573/D/SV/RO – Mid-Term Evaluation of theSpecial Pre-Accession Program for Agriculture and Rural Development inRomania – SAPARD, financed by Measure 4.2 – Technical Assistancewithin SAPARD Program. The objective of the Project was the quality andefficiency evaluation of the SAPARD Program, in the period 2000–2003,with the aim of its improvement.The performing of mid term evaluation waslimited to the analisys of initial elements of SAPARD programming andTERMS OF REFERENCE - Updating of Mid-Term Evaluation of the SAPARD Program 10

implemention in the previous period, including all kind of outputs, results,impact indicators form the point of view of efectiveness, eficiency andcorrectness of the action carried out. A summary of Mid-Term EvaluationReport, including the observations and recommandations, was presentedin the framework of the SAPARD Program Monitoring Committee in July2004.2. CONTRACT OBJECTIVESThe Mid-Term Evaluation Update aim is the examination and analysis ofresults produced by the SAPARD program, taking into accountrecommendations and observations contained in the already carried outMid-Term Evaluation. This would contribute to the improvement of thecurrent implementation stage of the Program and monitoring system, andto the eventual re-orientation and adjustment of budget allocations amongthe Program Measures in its last year of implementation.2.1 General Objectives:Updating of Mid-Term Evaluation general objective is the examination andanalysis of SAPARD Program results in 2000-2005, also with reference tothe recommendations and observations contained in the Mid-TermEvaluation Report concerning the implementation period 2000-2003, withreference to the two major objective of the Program as defined in Article 1(2) of the EC Regulation 1268/1999, as well as the provisions of Regulation445/2002, Article 57(1).In particular the mid-term updated evaluation must supply information on theimplementation and impacts of SAPARD Program, with the aim of:1) increasing the accountability and transparency of the programimplementation, with regard to the all actors involved, and in particularthe administrative and budget authorities and the public;2) improving the implementation of the program by contributing to informedplanning and decision-taking concerning needs, delivery mechanismsand resources allocations.2.2 Specific Objectives:The role of SAPARD Program Mid-Term Evaluation Updating is to improvethe previous evaluation, ensuring necessary corrections and eliminating anyconfusions and information gaps, aiming at displaying the actual state of theTERMS OF REFERENCE - Updating of Mid-Term Evaluation of the SAPARD Program 11

implementation and of its effects and at drawing up functional proposals toput in practice past and future valuable recommendations.The specific objective of the update is to improve the SAPARD Programimplementation, to propose and discuss with the Contracting Authorityreadjustments of the budget allocations for each Measure and, if the case,reorientations of them, in terms of strategic approach and having in view thefuture programming objectives.This updating should be carried out in consideration of the outputs andresults of Program implementation from the point of view of efficiency,effectiveness and quality of implementation as well as the adequacy andconsistency of objectives established in NAPRD.In more details, the Update Mid Term Evaluation should assess keyaspects of the assistance as relevance, effectiveness, efficiency, utility andsustainability, through:• The analysis of:- the correlation between the Program priorities and the proposedstrategy and objectives and the compatibility between the general, specificand operational objectives, as well as the utility, relevance, coherence,efficiency, effectiveness and durability of outcomes in relation to the realneeds of development of agriculture and rural areas taking into account thepresent and future context, in view of proper adjustment of the Program.- the adequacy of the strategy contained in the Program and theconsistency of its implementation taking into consideration themodifications occurred in the current situation in comparison with theelaboration and inception of the Program. This exam could highlight thenecessity of modifications concerning strategies and budgetary allocationsamong different measures.- the efficiency and effectiveness of Program implementation takinginto account the objectives quantified at operational (outputs), specific(results) and general (impact) level.- the results of the previous evaluation (for example: the validity of theanalysis in relation to imbalances and gaps that the Program is designedto approach; the validity and relevance of objectives compared with theneeds, as well as the coherence between the objectives of the SAPARDassistance and the objectives of other forms of national and external publicintervention which can influence the Program; the identification of newTERMS OF REFERENCE - Updating of Mid-Term Evaluation of the SAPARD Program 12

factors or other modifications which could have an effect on needs,strategies, efficiency and effectiveness of Program implementation).- the validity of assessment indicators referred to program objectivesmonitoring and evaluation.- the extension of horizontal issues (equal opportunities for men andwomen, conservation of natural patrimony, environmental protection, Stateaid) integration in the Program assistance.- the quality of Program implementation, mechanisms used to that end,the legal and administrative structure created for its implementation (theclear definition of responsibilities of authorities and institutions involved inthe Program implementation and monitoring, the quality of the monitoringsystem, the quality, transmission method, transparency and frequency ofinformation collection and availability for the monitoring system, proceduresand criteria of project selection).- the intervention policy ability to produce results lasting in the longterm, and after the end of the program activities.• Answer to the applicable common evaluation questions as set inAnnex 1, and elaborate substantial conclusions as regards: the overalleffects of SAPARD on the implementation of CAP and on solving thepriority and specific problems for the sustainable adaptation of agriculturalsector and rural areas in Romania.• Present the intermediate effective outputs (achieved investments,outcomes of the achieved investments compared with the values plannedin the measures).3. CONSIDERATIONS AND RISKS3.1 Considerations emphasizing the necessity of the project:The evaluator shall be supported in the implementation of his tasks by theManaging Authority and the SAPARD Agency. All available informationrelevant to the evaluation shall be made accessible for the evaluator.The successful completion of the project is subject to the following keyassumptions:• Sufficient level of cooperation with information sources outside theManaging Authority and the SAPARD Agency (other PublicTERMS OF REFERENCE - Updating of Mid-Term Evaluation of the SAPARD Program 13

Administrations, National Institute for Statistics, Specific Universities,etc…);• Good cooperation and support from social and economic partners.3.2 Risks:The major risks of the project can be attributed to the uniqueness ofSAPARD. Needed statistical data may not be available or may not bepresented in the appropriate manner (format) to allow the respectiveanalyses. This issue must be addressed in the Inception Report at theproject start up. This report must specify the data collection strategy andthe analysis methodology to be followed.4. FIELDS OF ACTIVITY4.1. Geographic scope and time frameThe mid-term evaluation update should cover the period 2000-2005 andthe “geographical area” of the NPARD.4.2 Regulatory ScopeThe mid-term evaluation update has as object the whole implementation ofthe National Program for Agriculture and Rural Development for 2000-2006set up within the framework of Regulation (EC) 1268/1999 and Regulation(EC) 445/2002. The updating of mid-term evaluation should take intoconsideration the implementation progress of the measures accredited in2002 and 2004 during the period 2003-2005, not covered by the mid-termevaluation for 2000-2003 (respectively Measures 1.1, 2.1, 4.2 andMeasures 3.1, 3.4 and 4.1). The measures of the Program which are notyet operational must anyway be subject to evaluation in the context of thereview of the ex-ante evaluation following the Commission’s guidelines forthe ex-ante evaluation: “The New Programming Period 2000-2006: the Exanteevaluation of the Structural Funds Interventions”. The present Mid-Term Evaluation Updating should give more details concerning themeasures which did not constitute the object of the previous mid-termevaluation, respectively the measures 1.2, 2.2, 3.2, 3.3 and 3.5.4.3 General and Specific obligatory ActivitiesThe mid-term evaluation update should comprise:TERMS OF REFERENCE - Updating of Mid-Term Evaluation of the SAPARD Program 14

• the exam of ex-ante evaluation validity;• the strategic evaluation (exam of effective implementation of thestrategy through the implementation of measures and submeasures);• utility, relevance and coherence of specific objectives in relation withthe general objectives (impact);• the evaluation of effectiveness, efficiency and durability of outputsand results.• The analysis of all the measures whose implementation has alreadystarted by the time of the present contract signature;• The analysis of the measures whose implementation has not yetstarted by the time of the present contract signature, but which are tobe implemented, in accordance with the requirements and guidelinesprovided for the ex-ante evaluation of the SAPARD Program.• A particular emphasis should be putted on the evaluation ofmeasures that have not been evaluated by the initial Mid-TermEvaluation Report.• Analysis of acquis communautaire implementation through SAPARDProgram.The update of the mid-term evaluation shall include an analysis andappropriate comparison with the results achieved with the implementationof the pilot project under Special Preparatory Program (SPP) for thepreparation of SAPARD implementation.4.3.1 Mid-Term Evaluation Updating StepsStarting from the Mid-Term Evaluation carried out with reference to theimplementation of the SAPARD Program in Romania, for the period 2000-2003, the evaluator should develop the mid-term evaluation updatingthrough a process including the following activities:• Structuring – the set of common evaluation questions in Annex 1clarifies the effects to be evaluated, defining the relevant criteria andindicators.• Data collection – The Contracting Authority shall provide to theevaluator all available monitoring data as well as access to thenecessary data from the SAPARD Agency. In case the above dataare not sufficient to answer properly the evaluation questions or toquantify the evaluation indicators as set out in the SAPARDTERMS OF REFERENCE - Updating of Mid-Term Evaluation of the SAPARD Program 15

operational measures, the evaluator must collect additional relevantdata by appropriate means, including, where necessary, studies bysub-contracting with relevant and competent research andinformation providers.• Analysis – the evaluator must process and compare the data as wellas assess the Program effects and quality of its implementation andmonitoring.• Judgement – the evaluator must draw substantiated conclusionsbased on the above analysis. This judgement should concern theimpact, result and outputs of SAPARD Program implementation, aswell as the validity of indicators and recommendations provided bythe initial mid-term evaluation.• Reporting - the evaluator shall prepare both the versions of mid-termevaluation update report and final mid-term evaluation update report,as described at point 7.3. Also, the evaluator will respect thecommon structure described below and will take into particularconsideration the specific measures that have not yet been the objectof the Mid-Term Evaluation carried out with reference to theimplementation of the SAPARD Program in Romania, for the period2000-2003.4.3.2 Obligatory elements of the Mid-Term Evaluation Updating4.3.2.1 Analysis of the ex-ante Evaluation and the Mid-TermEvaluation 2000-2003 resultsFirst of all, the mid-term evaluation updating should carry out the analysisof the ex-ante evaluation results, as well as of the conclusions andrecommendations of Mid-Term Evaluation Report for 2000-2003.This analysis will answer to the following questions:• Whether the strategy concerning the support under the SAPARDProgram is adequate and consistent with the general objectives ofNPARD and, if not, the needed changes.• Whether the mechanisms for the implementation of the strategy areeffective and efficient.• Whether certain factors or conditions which can influence negativelythe Program implementation or its efficiency and effectiveness doexist and which could be the possible solutions.TERMS OF REFERENCE - Updating of Mid-Term Evaluation of the SAPARD Program 16

• Whether the Mid-Term Evaluation Report for 2000-2003 results areadequate to the present situation of program implementation and, ifnot, which are the reason and in which way the mid-term evaluationupdate will answer to the present situation needs.• Any other relevant information for the monitoring and evaluation ofthe Program.4.3.2.2 The validity of SWOT analysis carried out during the Program’sex-ante EvaluationA key element of the mid-term evaluation update is represented by theSWOT analysis having in view that it has provided the base for theelaboration of the strategy contained in NPARD. Within the ex-anteevaluation this analysis was checked including the adequacy of identifiedopportunities and risks.The mid-term evaluation update must verify if this analysis is still valid.During the mid-term evaluation update, the last evolutions and trends in theeconomic and social sector, and particularly in agriculture and rural areas,should be examined in view of identifying the changes occurred and theircauses.Also, both the priorities for the achievement of the general objectives inNPARD and their conformity with the identified opportunities and needs aswell as the risk factors should be re-examined.The updating of the mid-term evaluation should consider the past andcurrent public interventions, including Programs carried out by otherorganizations, in order to verify to what extent the Program priorities, asthey were identified by the SWOT analysis in the NPARD, correspond tothe present actual needs.Within this section of the mid-term evaluation update, the SWOT analysisand the contribution of the ex-ante evaluation should be re-viewed. Thevalidity of the ex-ante analysis should be checked upon and, whereappropriate, recommendations should be made for its updating.This section of the mid-term evaluation update should be relatively short,especially if no major changes of the initial conditions have occurred or noTERMS OF REFERENCE - Updating of Mid-Term Evaluation of the SAPARD Program 17

elevant information has been identified, both leading to a significantmodification of the preliminary analysis.In the Conclusions of this section of the mid-term evaluation update thefollowing elements should be presented:‣ Evaluation of the validity as regards the priority and major problemswhich must be solved as well as the necessity to adopt certainchanges in the conclusions of the analysis;‣ Evaluation of correlation between the strategic objectives and theidentified problems;‣ Any other information or factors which facilitate the economic andsocial cohesion, the protection of environment and equalopportunities for men and women as well as the implementation ofthe acquis communautaire.4.3.2.3 Evaluation of adequacy and consistency of the DevelopmentStrategy in NPARDOn the basis of the economic and social problems identified and theirrelations to the major needs, the mid-term evaluation update should carryout the examination of the balance and appropriate combination betweenthe priorities established in NPARD and the support measures provided bythe SAPARD Program.To that end, the mid-term evaluation update should verify the contributionof each specific priority objective to the achievement of the generalobjectives.In the end of this section, a revised concept of the Program should bepresented where appropriate. This can be determined either by theimproper ex-ante/initial mid-term evaluation or by omissions of itsrecommendations as well as by changes of the needs and major problemsduring the Program implementation which imply the re-evaluation of theconsistency of the development strategy, measures and priorities.The mid-term evaluation update must furnish to the Managing Authority forthe SAPARD Program a clear motivation of the validity and weight of eachmeasure within the general objectives.The mid-term evaluation update should analyse the inter-connectionbetween general, specific and operational objectives.TERMS OF REFERENCE - Updating of Mid-Term Evaluation of the SAPARD Program 18

4.3.2.4 The Quantification of Goals – Outputs, Results and ImpactThe mid-term evaluation update should consider the quantification of theoperational objectives for the program and for each measure, taking intoaccount that the achievement of the operational objectives contributes tothe achievement of the specific and general objectives of the Program.The mid-term evaluation update should examine the quantificationeffectiveness of measures operational objectives.The mid-term evaluation update should include an analysis of the logicalconnections between the quantified operational objectives of the measuresand the general and specific objectives of the Program in view of ensuringthe economic and social cohesion.Also, the mid-term evaluation update should analyse the implementationindicators, the result and impact of each measure and sub-measure andshould propose their improvement and adaptation to the present nationalcontext.The common evaluation questions with associated criteria and indicators,as proposed in the Commission evaluation guidelines for SAPARD anddefined in Annex 1 of these Terms of Reference, must be adapted to thecontext of this evaluation update. It has to be established whether allquestions, criteria and indicators are relevant and how will they be appliedin each case. If this is not the case, a justification of their non-applicabilityshould be given.The indicators must reflect the inter-dependence between the outcomes ofthe proper implementation of the Program measures. Also, the mid-termevaluation update must identify every indicator which is strongly influencedby external factors and propose improvement or replacement solutions.To that effect, the mid-term evaluation update should comprise:‣ Analysis of output, results and impact indicators which quantify theobjectives and the major identified imbalances;‣ Verification of output, results and impact indicators correlation withgeneral, specific and operational objectives;‣ Evaluation of output, results and impact indicators in the context ofmonitoring of equal opportunities for men and women, environmentprotection and other horizontal issues;TERMS OF REFERENCE - Updating of Mid-Term Evaluation of the SAPARD Program 19

‣ Analysis of output, results and impact indicators applicability basedon their capacity to furnish a precise image of the implementation,monitoring and evaluation of the Program.The mid-term evaluation up-date should supply information concerning thecurrent stage of the acquis communautaire implementation, as well as theagriculture and rural development improvements through SAPARDProgram in Romania.The mid-term evaluation update must also furnish information concerningthe contribution of measures objectives to equal opportunities for men andwomen and environment protection, as well as to the implementation of theacquis communautaire.4.3.2.5 Evaluation of the effectiveness and efficiency of ProgramImplementation – Evaluation of policy and distribution ofresourcesThe updating of mid-term evaluation has the role to formulate conclusionsconcerning the effectiveness and efficiency of the Program implementation,as a whole and by measure, taking into account the intermediate outputsand results and their impact with reference to the activities, specific andgeneral objectives.Based on the formulated conclusions the mid-term evaluation updateshould make results forecasts. The operational measures should be usedas base unit of the analysis, which means that the analysis should answerthe corresponding general and specific evaluation questions.Efficiency – inputs, compared with outputs and resultsThe updating of mid-term evaluation must examine the efficiency of theProgram implementation from the point of view of existing relationsbetween relevant outputs and inputs needed to produce them.Firstly, the analysis should identify the average costs per unit (outputscompared with inputs). Information regarding costs per unit should becompared with regional, national and European values in the samecategory.Evaluation of efficiency should be made also with regard to the “deadweight”effects (modifications in the situation of beneficiary which wouldTERMS OF REFERENCE - Updating of Mid-Term Evaluation of the SAPARD Program 20

have occurred even without public financing), “replacement” effect (effectobtained in a certain geographic area to the detriment of another one), and“leverage” (the fact that public financing implies also beneficiary’s ownexpenditure).Effectiveness – outputs and results compared with specific andoperational objectivesThe mid-term evaluation update should examine the implementationeffectiveness starting from the monitoring indicators established in theProgram.The analysis of the extent of specific and operational objectivesachievement should be made as follows:o Analysis of progresses registered in achieving the operational objectivesshould be based on the effective outputs of the Program for eachmeasure, compared with the value of preliminary indicators establishedin the respective measure.o Analysis of progresses in achieving the specific objectives should bebased on the results pertaining to each priority axis established in theProgram.o Analysis of progresses in achieving the operational and specificobjectives should be based on current application stage of theobservations and recommendation of the Mid-Term Evaluation Reportfor the period 2000-2003.Following the analysis execution, the mid-term evaluation update shouldpresent the conclusions concerning the effectiveness of the Program.Impact - global objectivesWhen evaluating the effectiveness and the impact of the measuresintended for rural development, the relevant common evaluation questionsshould be utilized together with the relevant criteria and indicators toestablish the actual contribution (added value) of the SAPARD Program inRomania.The mid-term evaluation update must present conclusions concerning theadequacy of the development strategy provided in NPARD and thedistribution of financial resources among priorities in order to establish theTERMS OF REFERENCE - Updating of Mid-Term Evaluation of the SAPARD Program 21

impact of the Program in relation to the identified needs. Conclusionsshould be based on the analysis of the effectiveness and efficiencyregistered in the achievement of Program objectives. Conclusionsregarding the inadequate weight of priorities or measures in the Programshould constitute the basis for recommendations concerning the changesneeded in the structure of the financial support.4.3.2.6 Quality of Program implementation and organization ofMonitoringThe impact of the policy is directly determined by the management andfunctioning capacity of the implementing bodies.The mid-term evaluation update must examine the quality and efficiency ofimplementation, management and monitoring mechanisms taking intoconsideration that any weakness can negatively influence the impact offinancial assistance provided by the Program.The quality of Program implementation and monitoring should be examinedby:o Evaluation of transparency and clear separation of responsibilities inthe management and monitoring of the Program implementation;o Evaluation of promotion activities and level of Program knowledge bythe implementation bodies;o Evaluation of control mechanisms based on the audit reports andirregularities ascertained;o Analysis of projects eligibility and selection criteria in view of ensuringthe quality of their implementation which should be reflected in theProgram objectives;o Verification of existence of flexible, transparent procedures andcriteria for the selection of projects so that the achievement ofProgram objectives and utilization of available financial resourcesshould be efficiently ensured;o Evaluation of the manner in which the economic and social partnerscontribute to the quality of Program implementation;o Evaluation of the harmonization level of national legislation to theacquis communautaire as well as its implementation degree,following the execution of the SAPARD Program in Romania;o Evaluation of the current administrative system capacity to implementthe SAPARD Program and the acquis communautaire related toTERMS OF REFERENCE - Updating of Mid-Term Evaluation of the SAPARD Program 22

Common Agricultural Policy, taking into consideration the newmeasures which follow to be accredited in 2005.This analysis must be based, where appropriate, on the answers to therelevant cross-cutting evaluation questions as set out in annex 1;4.3.2.7 Institutional impact• Examining the progress achieved in building the administrativecapacity needed for the implementation of the acquis communautaireconcerning the CAP, as well as changes in the management andorganization of relevant public policies in Romania;• Assessing the degree to which the national legislation supporting theimplementation of the Program is developed and harmonized with theacquis communautaire.The analysis must be based, where appropriate, on the responses to therelevant cross-cutting evaluation questions as set out in annex 1.4.3.2.8 Conclusions and recommendationsEach section of the mid-term evaluation update should present conclusionsconcerning the weaknesses of SAPARD Program implementation inRomania and recommendations for its improvement. This, starting from theanalysis of the Mid-Term Evaluation Report for the period 2000-2003 andfrom the examination of its results with reference to indicators used andrecommendations proposed.Conclusions and recommendations should be included in the mid-termevaluation update and should refer to the following aspects:1. Examination of the initial mid-term evaluation• applicability of recommendations by Public Administration and otherrelevant organisms;• validity (completeness, adequateness and suitability) of output,results and impact indicators proposed by the initial Mid-TermEvaluation Report.2. Analysis of the ex-ante evaluation• Adequacy of the existing strategy and the necessity to complete orchange it;TERMS OF REFERENCE - Updating of Mid-Term Evaluation of the SAPARD Program 23

• Effectiveness of instruments for the implementation of the CommonAgricultural Policy;• Risk factors which influence the effectiveness and efficiency ofimplementation;• Relevant information concerning possibilities of evaluation andmonitoring.3. SWOT validity• Validity of priorities and major imbalances which have to beovercome and/or introduction of necessary changes;• Correlation between the objectives and identified needs;• Factors which facilitate the economic and social cohesion,environment protection and equal opportunities for men and women.4. Evaluation of adequacy and consistency of the development strategyin NPARD• Present validity of the global strategy concept and structure;• Justification of the share and weight of each priority axis;• Program coherence – correspondence between Program objectives,NPARD and the National Development Plan objectives, in view ofensuring the achievement of economic and social cohesion as well asthe correspondence between national policies and priorities and thecommunity ones;• Correspondence between operational, specific and generalobjectives.5. Quantification of goals – outputs, results and impact• Adequacy of indicators used to quantify the objectives andimbalances;• Adequacy of indicators concerning the general, specific andoperational objectives;• Adequacy of indicators which monitor the impact on the equalopportunities for men and women, environment protection and otherhorizontal issues;• Adequacy and timetable of data collecting procedures;• Applicability and utility of indicators used to obtain a correct andtimely image of the efficiency of Program implementation, monitoringand evaluation.6. Evaluation of effectiveness and efficiency of Program implementationTERMS OF REFERENCE - Updating of Mid-Term Evaluation of the SAPARD Program 24

• Results and progresses registered in achieving the objectives;• Efficient and sufficient information concerning beneficiaries.• Financial share of each priority based on the first results obtained andthe resulted impact compared with that foreseen and, in case ofimproper results, the needed changes;• Efficiency of outputs and results compared with the utilization offinancial resources;• Effectiveness of horizontal measures implementation with regard tothe equal opportunities for men and women and the environmentprotection;• Progresses registered in achieving the general, specific andoperational objectives;• Correspondence between the foreseen and effective impact ofProgram implementation for each priority axis and measure;• Correspondence between the first results obtained and financialallocations for each priority and measure within the Program;• Correspondence between allocated resources and obtained outputsand results.7. Quality of implementation and monitoring management• Effectiveness and efficiency of Program implementation andmanagement;• Transparent and competitive procedures for project selection;• Actual accountability according to the requirements of the relevantnational and community legislation;• Contribution of economic and social partners.4.3.2.9 Structure of Mid-Term Evaluation Updating ReportThe mid-term evaluation updating Report must describe comprehensivelythe evaluated Program, including its context and scope, procedures andresults of evaluation as well as the conclusions and recommendations.The structure of the mid-term evaluation update Report to be presented tothe European Commission (Regulation EC 1750/1999) is the following:TERMS OF REFERENCE - Updating of Mid-Term Evaluation of the SAPARD Program 25

a) Executive summaryThe executive summary should contain the main results and conclusions ofthe mid-term evaluation update. It is preferably that the executive summarydoes not exceed 5 pages.b) IntroductionThe introduction should present general and contextual informationconcerning the Program: specific national policies, economic and socialneeds justifying the assistance, definition of beneficiaries or target groups.Also, the Introduction should furnish information on actions previouslyimplemented. To that effect, it should include key elements (updated) oftheir implementation - parties involved, institutional framework, period,general financial information, a short description of priorities and measurestaken into account for evaluation.At the same time, the Introduction should furnish information on theevaluation process: presentation of the Terms of Reference, scope andobjectives of the evaluation, common and Program specific evaluationquestions.It should present a succinct description of previous evaluations relevant tothe Program.c) Methodological aspectsIn this section the logic of the methodological framework and itsconsequences should be presented. Also, it should be presented thegeneral evaluation system and methods used during the evaluationprocess:- data sources, data collecting methodology (questionnaires,interviews, size and selection criteria for samples), informationregarding the calculation method for the evaluation of data quality andcorrectness and the identification of possible inaccuracies;- methodology applied to answer to the evaluation questions and toissue the conclusions.The Report should clarify any problem or limitations in relation to theapplication of this methodology.d) Presentation and analysis of collected informationTERMS OF REFERENCE - Updating of Mid-Term Evaluation of the SAPARD Program 26

applied by the institutions involved in the implementation and managementof the Program and in the future rural development programming period.This section should clearly answer to the evaluation questions dulyreasoned on the basis of the evaluation work, to all the evaluationquestions proposed in the specifications as well as any other questiondefined during the structuring phase of the evaluation. The conclusion partof each question must follow on directly from the analysis and include areference to the judgement criterion.f) AnnexesThe annexes should contain detailed information in relation to the Terms ofReference, complete sets of data, analytical details, detailed monographsand the structure of questionnaires.An additional but not compulsory section which can contribute to theevaluation credibility is that dedicated to opinions from the economic andsocial partners as regards the quality of achieved and presentedevaluation.4.3.2.10 BibliographyIn the preparation of the mid-term evaluation update, the followingdocuments must be used:• “Guidelines for the Evaluation of Rural Development ProgrammesSupported by SAPARD” (Doc. DG-Agri, 04.2001);• “Guidelines for the mid-term evaluation of rural developmentPrograms funded by SAPARD 2000-2006” (Doc. DG-AGRI 09/2002);• “The National Program for Agriculture and Rural Development”approved by the European Commission on 12 December 2000, asmodified by European Commission Decision No H/2002/1936 on 12of July 2002, European Commission Decision on 27 th of May 2003,European Commission Decision on 1 st of August 2003 and EuropeanCommission Decision on 17th September 2004.• “Multi-Annual Financing Agreement” for the Special AccessionProgramme for Agriculture and Rural Development in Romania,between the Commission of the European Communities on behalf ofthe European Community and Romania, signed in Brussels on 2 nd ofFebruary 2001 and ratified by Romanian Law no. 316/2001 (OfficialJournal of Romania no. 362/2001);TERMS OF REFERENCE - Updating of Mid-Term Evaluation of the SAPARD Program 28

• “Annual Financing Agreement 2000” for the Special AccessionProgramme for Agriculture and Rural Development in Romania,between the Commission of the European Communities on behalf ofthe European Community and Romania, signed in Brussels on 27 th ofFebruary 2001 and ratified by Romanian Law no. 317/2001 (OfficialJournal of Romania no. 338/2001);• “Annual Financing Agreement 2001” for the Special AccessionProgramme for Agriculture and Rural Development in Romania,between the Commission of the European Communities on behalf ofthe European Community and Romania, signed in Brussels on 30 th ofJanuary 2002 and ratified by Romanian Law no. 416/2002 (OfficialJournal of Romania no. 508/2002);• “Annual Financing Agreement 2002” for the Special AccessionProgramme for Agriculture and Rural Development in Romania,between the Commission of the European Communities on behalf ofthe European Community and Romania, signed in Bucharest andBrussels on 1 st of April 2003 and ratified by Romanian GovernmentUrgency Ordinance no. 26/2003 (Official Journal of Romania no.283/2003);• “Annual Financing Agreement 2003” for the Special AccessionProgramme for Agriculture and Rural Development in Romania,between the Commission of the European Communities on behalf ofthe European Community and Romania, signed in Brussels andBucharest on 31 st of July 2003 and ratified by Romanian Law no.496/2003 (Official Journal of Romania no. 843/2003);• Mid-Term Evaluation of SAPARD in Romania for the period 2000-2003;• Annual SAPARD Implementation Reports.The following documents are recommended:• Working documents of SAPARD Agency, elaborated on the basis ofthe implementation procedures;• “The evaluation of socio-economic Programs; indicators for themonitoring and the evaluation of Programs (MEANS, Vol.2). Thiswork approaches methodological aspects referring to the definition ofTERMS OF REFERENCE - Updating of Mid-Term Evaluation of the SAPARD Program 29

indicators and furnishes the list of indicators for the main fields ofintervention of the Structural Funds;• “Guidelines for Evaluation of rural development Programs 2000-2006supported from the European Agricultural Guidance and GuaranteeFund” (Doc. DG AGRI VI/8865/1999);• “Guidelines for the mid term evaluation of rural developmentPrograms 2000-2006 supported by the European AgriculturalGuidance and Guarantee Fund” (DOC STAR VI/43517/2002);• Council Regulation (EC) No.1268/1999 of 21 June 1999 onCommunity support for pre-accession measures for agriculture andrural development in the applicant countries of central and easternEurope in the pre-accession period;• Council Regulation (EC) No 1260/1999 of 21 June 1999 laying downgeneral provisions on the Structural Funds;• Commission Regulation (EC) No. 2759/1999 of 22 December 1999laying down rules for the application of the Council Regulation (EC)No.1268/1999 on Community support for pre-accession measures foragriculture and rural development in the applicant countries of centraland eastern Europe in the pre-accession period;• Commission Regulation (EC) No. 2222/2000 laying down financialrules for the application of Council Regulation (EC) No.1268/1999 onCommunity support for pre-accession measures for agriculture andrural development in the applicant countries of central and easternEurope in the pre-accession period;• Council Regulation (EC) No 2008/2004 of 16 November 2004amending Regulation (EC) No 1268/1999 on Community support forpre-accession measures for agriculture and rural development in theapplicant countries of central and eastern Europe in the preaccessionperiod;• Council Regulation (EC) No 1257/1999 of 17 May 1999 on supportfor rural development from the European Agricultural Guidance andGuarantee Fund (EAGGF) and amending and repealing certainRegulations;• Commission Regulation (Ec) No 1750/1999 of 23 July 1999 layingdown detailed rules for the application of Council Regulation (EC) No1257/1999 on support for rural development from the EuropeanAgricultural Guidance and Guarantee Fund (EAGGF);• Commission Regulation (EC) No 445/2002 of 26 February 2002laying down detailed rules for the application of Council RegulationTERMS OF REFERENCE - Updating of Mid-Term Evaluation of the SAPARD Program 30

(EC) No 1257/1999 on support for rural development from theEuropean Agricultural Guidance and Guarantee Fund (EAGGF);• COM(2004)490, proposal for a Council Regulation on support forrural development by the European Agricultural Fund for RuralDevelopment (EAFRD) and the subsequent regulation whenapproved.4.4 Project management, contractor’s tasks and responsibilities:The direct beneficiary of the project and Contracting Authority responsiblefor the management of the project is the Managing Authority within theMinistry of Agriculture, Forests and Rural Development (MAFRD).SAPARD Agency have to approve the documents concerning the project,as mentioned in the Manual of Procedure for Measure 4.2 „TechnicalAssistance” and is responsible of the execution of the payments afterauthorization of payment requests, in compliance with all major proceduresapplied for SAPARD.Obligations of the Contracting Authority in RomaniaThe Contracting Authority has the following obligations:• To ensure a good cooperation and communication of the ContractingAuthority relevant personnel with the Contractor staff during thewhole duration of the Evaluator activity in relation to the mid-termevaluation update;• To support the Evaluator in the technical and logistic organization ofthe evaluation process;• To grant the necessary support to the evaluation team experts;• To ensure and to facilitate the access to available data and nationallegislation for the correct carrying out of the evaluation;• To be fully available to promptly answer the evaluator requests forhelp in the solution of critical situations that limit or embed the correctfulfilment of its functions, especially with reference to the collectionand processing of information provided in the territory;• To follow up the correct utilization of the funds for the carrying out ofevaluation;• To submit the final mid-term evaluation update Report to theEuropean Commission after having submitted its synthesis andconclusions to the monitoring committee.TERMS OF REFERENCE - Updating of Mid-Term Evaluation of the SAPARD Program 31

Obligations of the ContractorThe Contractor has the responsibility:• To ensure all the equipment, endowments, material goods andservices, as well as logistic support needed to perform the mid-termevaluation update;• To ensure the financial and administrative management necessaryfor the carrying out of the mid-term evaluation update;• To provide the qualified key experts, as specified at point 6.3 of theseToR, as well as other needed supporting staff, being responsible forfees, lodging, international and local travel of is international experts;• To organize the visits of Romanian and foreign experts in the countryand outside the Country;• To ensure the correct utilization of the funds for the carrying out of themid-term evaluation update according to the contract signed with theContracting Authority;• To start the project activities on the first day after the contractsignature, as provided by point 5.2 of these ToR;• To submit to the Contracting Authority detailed Reports on activitiesdeployed to that effect, as provided at point 7 of these ToR;• To report to the Contracting Authority on the activities deployed andeventual obstacles met in view of adoption of operative remedialmeasures;• To identify the relevant quantitative and qualitative indicators ofoutput, results and impact, taking into account the indicatorsformulated in the Mid-Term Evaluation Report for 2000-2003, inNPARD and in the Guidelines of the Commission.• To collect the necessary available data and to elaborate and collectother relevant data (interviews, questionnaires, etc…) necessary tothe evaluation of identified relevant indicators;• To process the collected data and indicators, including thecorresponding results in the mid-term report update.• To ensure the application of the ToR, with reference to the contentand structure of the mid-term evaluation update report.• To provide the Contracting Authority with one Inception Report, oneInterim Activity Report and one Final Activity Report, as described atpoint 7.1 and 7.2;• To provide the Contracting Authority with the first version of the Mid-Term Evaluation Updating Report, the draft final version of the Mid-TERMS OF REFERENCE - Updating of Mid-Term Evaluation of the SAPARD Program 32

Term Evaluation Updating Report and the Final Mid-Term EvaluationUpdating Report, as described at point 7.3 of these Terms ofReference. These reports must be characterised by pertinent andcredible results, judgements and recommendations. Each one ofthese reports must respect the structure described at point 4.3.2.9 ofthese ToR;• To respect indications formulated, in maximum 15 days since thesubmission of the Inception Report and the activities Reports, by theContracting Authority;• To consult with the Contracting Authority regarding the first version ofthe Mid-Term Evaluation Report, taking into account its observationsand recommendations;• To consult with the Contracting Authority, during the elaboration ofthe Mid-Term Evaluation Updating, with reference to the EuropeanCommission remarks concerning the Mid-Term Evaluation Report forSAPARD Program during the period 2000-2003;• To incorporate modifications recommended by the ManagingAuthority, the Monitoring Committee and the European Commissionin the mid-term evaluation update Report, as described at point 5.2 ofthese Terms of Reference;• To ensure the finalisation, in the time and conditions provided for bythese ToR, of the reports mentioned at point 7 (on paper and onelectronic format, CD/floppy, in Romanian and English);• To ensure his own independence and the independence of all hisworking staff, as described at point 4.4 of these ToR.In the fulfilment of their responsibilities both the Contracting Authority andthe Contractor shall respect the “ethics clauses" as set in the Manual forthe implementation of Measure 4.2 “Technical Assistance”.Awarding criteriaOn receiving tenders, the representatives of the contracting authority willrecord them and provide a registration number for those delivered by hand.Only tenders contained in envelopes received by the date and timeindicated in the tender dossier will be considered for evaluation.TERMS OF REFERENCE - Updating of Mid-Term Evaluation of the SAPARD Program 33

Initially, only the technical offers will be opened. The sealed envelopescontaining the financial offers will be retained by the Contracting Authorityonce signed by the President and the Secretary of the EvaluationCommittee.The Evaluation Committee checks the administrative compliance of tenderswith the instructions given in the tender dossier. Any formal errors or majorrestrictions affecting performance of the contract or distorting competitionwill result in the rejection of the tender concerned.The quality of each technical offer shall be evaluated in accordance withthe award criteria and the associated weighting as detailed in theevaluation grid included in this tender dossier. No other award criteria willbe used. The award criteria will be examined in accordance with therequirements as indicated in the Terms of Reference.When evaluating technical offers, each member awards each offer a scoreout of a maximum 100 points in accordance with the technical evaluationgrid (setting out the technical criteria, sub-criteria and weightings) laid downin the tender dossier.Where the content of a tender is incomplete or deviates substantially fromone or more of the technical award criteria laid down in the tender dossier,the tender is automatically rejected and no points are awarded.Besides the awarded numerical score, each member of the EvaluationCommittee must explain the reasons for his choice and defend his scoresbefore the Evaluation Committee. The Committee discusses each technicaloffer and each member awards it a final score. The aggregate final score isthe arithmetic average of the individual scores. Once the EvaluationCommittee has established the final score of each technical offer (thearithmetic average of the scores awarded by each member), any tenderfalling under the 80-point threshold is automatically rejected. If no tenderachieves 80 points or more, the tender procedure is cancelled. Thecommittee considers only tenders that have obtained at least 80 points. Ofthese tenders, the best technical offer is then awarded 100 points. Theothers receive points calculated using the following formula:Points = (initial score of the tender in question/initial score of the besttechnical offer) x 100.TERMS OF REFERENCE - Updating of Mid-Term Evaluation of the SAPARD Program 34

Upon completion of the technical evaluation, the envelopes containing thefinancial offers for tenders which were not eliminated during the technicalevaluation are opened and signed by the President and the Secretary ofthe Evaluation Committee at the session. At the session, the Committeechecks that the financial offers contain no arithmetic errors. Any arithmeticerrors which do not affect the offer content are corrected without prejudiceto the candidate.Comparison of the financial offers takes account of the total contract value.Financial offers exceeding the maximum budget allocated for the contractare eliminated.The lowest financial offer receives 100 points. The others are awardedpoints by means of the following formula:Points = (lowest financial offer/financial offer being considered) x 100.The most economically advantageous tender is established by weighingtechnical quality against price on an 80/20 basis. This is done bymultiplying:• the scores awarded to the technical offers by 0.80• the scores awarded to the financial offers by 0.20.The resulting technical and financial scores are then added together, andthe contract is awarded to the tender achieving the highest score.The entire procedure (technical and financial evaluation) is recorded inminutes to be signed by all members of the committee and approved by theContracting Authority.Evaluation of offers will be performed according to the procedure manualfor the implementation of Measure 4.2 „Technical Assistance”. The scoringof offers submitted by candidates will be performed on the basis of aevaluation grid in which the score of technical offer will represent 80%, andthat of financial offer 20% of the total score. The evaluation grid is shown inannex no. 2.Technical evaluation grid will contain the scoring for organisation andmethodology of technical offer, and the evaluation of the experts.TERMS OF REFERENCE - Updating of Mid-Term Evaluation of the SAPARD Program 35

Organisation and Methodology will be evaluated on the basis of therationale, strategy and programming proposed by the candidate. Thissection will represent the 30% of the technical evaluation final score.CVs of key experts, which are compulsory, will be evaluated on the basis ofqualifications and skills, general and specific professional experience. Thissection will represent the 70% of the technical evaluation final score.Financial offers will be evaluated on the basis of the total contract value,compared to the maximum budget available for the contract.The contract is financed through Measure 4.2 „Technical Assistance” withinSAPARD Program and is a global price contract, meaning that thecontractor will be paid only if the specified outcome is achieved in itscompleteness.Within the framework of the service contract, the financing could beeffectuated in two instalments, but only after the presentation of the Firstversion of the Mid-Term Evaluation Updating Report and the Final Mid-Term Evaluation Updating Report by the Contractor, and approval by theContracting Authority of the Reports mentioned at point 7 of these ToR. Noadvance payments are foreseen.Subcontracting within the framework of the contract is allowed. The valueof the subcontracted services must not exceed 30% of the total contractvalue.4.5 Independence of the EvaluatorThe evaluation shall be carried out by an independent Contractor (experts)in the sense of no direct involvement in the management, implementationand financing of the Program.The evaluator shall refrain from any relationship which would compromiseits independence or that of its personnel.The evaluator shall limit its role in connection with the project to theprovision of services.TERMS OF REFERENCE - Updating of Mid-Term Evaluation of the SAPARD Program 36

The evaluator and anyone working under its authority and control in theperformance of mid-term evaluation update shall be excluded from accessto EC financing available under the SAPARD Program.The experts must not have been direct beneficiary of the SAPARDProgram, nor should they have been involved in its management or in thedrafting of these terms of reference.Payments to the Evaluator under the project shall constitute the onlyincome or benefit it may derive in connection with the project and neitherhe/she nor his/her personnel shall accept any commission, discount,allowance, indirect payment or other consideration in connection with,relation to or in discharge of his/her obligation under the project.5. LOGISTICS AND PROGRAMMINGThe evaluation should involve the active participation of the relevantInstitutions (Ministry of Agriculture, Forests and Rural Development,Agriculture and Rural Development Counties’ Directorates, SAPARDAgencies with its BRIPS and other Institutions).5.1 Place where the project is executedProject activities will be executed in Bucharest and in the Counties, infunction of planned activities.The mid-term evaluation update must cover the geographic area of theNAPARD.The evaluator will participate to meeting and discussion not only on theRomanian territory but also outside Romania (consultations with theEuropean Commission in Brussels), on request.5.2 Period of the ProjectUpdating of mid-term evaluation should be completed in six months.The project activities must start on the first day after the contract signature.The updating should be carried out in three stages:1. the collection by the Evaluator of the necessary data (consultingdocuments, interviews, questionnaires, etc…), processing of theTERMS OF REFERENCE - Updating of Mid-Term Evaluation of the SAPARD Program 37

collected data, analysis of the results and preparation of the first versionof Mid-Term Evaluation updating Report, which is to be discussed withthe Contracting Authority coordinating the mid-term evaluation updateactivities, no later than 10 th of October 2005;2. the completion of the mid-term evaluation updating and submission ofthe draft final version of Mid-Term Evaluation Updating Report to theContracting Authority, no later than 10 th of November 2005, so thatenough time should be allocated for examination by the MonitoringCommittee and possible adjustment or completion of the Report;3. The Final Mid-Term Evaluation Updating Report must be submitted tothe Contracting Authority no later than 10 th of December 2005.In case the Managing Authority recommends modifications to the reportsmentioned above at points 1, 2 and 3, the evaluator must incorporate themand resubmit the respective reports in no more than 5 days since therequest.The Managing Authority will present the Final Mid-Term EvaluationUpdating Report to the European Commission no later than 31 st ofDecember 2005.In case the European Commission recommends modifications in the report,the contract will automatically be extended to include the evaluatorobligation of incorporating the recommended modification and resubmittingthe Update Mid-Term Evaluation Report in no more than 7 days since therequest by the Managing Authority.6. REQUIREMENTS6.1 General RequirementsSpecialised staff and staff-related expenses for missions inside and outsideRomania, equipments, endowments, goods and services concerning theimplementation of contract activities, and logistic support for the successfulaccomplishment of undertaken tasks are under complete responsibility ofthe services provider-contractor.6.2 Contractor Requirements- Companies with legal status, Romanian, foreign or mixed, which has asobject the activities requested through these Terms of Reference;TERMS OF REFERENCE - Updating of Mid-Term Evaluation of the SAPARD Program 38

- The company has to dispose of staff resources – at least 3 humanresources of the permanent staff, currently working in the relevantsectors for this contract, and should have suitable working space,suitable endowments, technical equipment and instruments necessaryto carry out the updating of the mid term evaluation report.- The company should present an activities portfolio showing experiencein projects and programs evaluation, in rural development sector;- The company (individual or consortium) annual turnover average, in thelast two operating years (2002, 2003) must exceed 1.000.000 EURO;- The Company balance sheet should demonstrate a profit for each oneof the last two years (2002, 2003);- The Company (individual or consortium) must have taken part to theimplementation of an international project of a nature and complexitysimilar to those of the present contract, in the last five years;- Credentials from at least one client who benefited of the Contractormonitoring and evaluation services.6.3 Key Experts RequirementsWorking staff for the contract should be composed of specialists inmonitoring, evaluation, agriculture, rural development and economics(minimum 8), autochthones or international, which could satisfy thefollowing requirements:6.3.1 Profile of International ExpertsKey expert 1 – Project Team Leader and Coordinator, expert I gradeQualifications and skills:- Economical or technical superior studies in the field of agri-food or ruraldevelopment;- Specialization studies in evaluation management and in the ruraldevelopment sector;- High proficiency in the English language, and very good knowledge oftechnical English concerning the specific sector.General professional experience- Adequate knowledge of the EU rural development policy, programssupported by Structural Funds as well as pre-accession instruments,especially SAPARD Program, for the countries in Central and EasternEurope.TERMS OF REFERENCE - Updating of Mid-Term Evaluation of the SAPARD Program 39

Specific professional experience- Experience of minimum 10 years as Team Leader in evaluation andmanagement of technical assistance projects or programmes;- Minimum 3 years experience as Team Leader in monitoring andevaluation of SAPARD Program and Structural Funds Programs, in thefield of agriculture and rural development;- Coordination experience in monitoring and evaluation – minimum 5years;- Number of projects on monitoring and evaluation of SAPARD Programand other programs financed by Structural Funds, in the field ofagriculture and rural development, in which the expert has worked asTeam Leader (minimum 2);- Adequate knowledge of rural development issues and of the socioeconomicsituation in the rural areas of Central and Eastern Europe.Key expert 2 – Expert in the Evaluation of Rural Development Programs,expert II gradeQualifications and skills- Economical or technical superior studies in the field of agri-food or ruraldevelopment;- Specialization studies in monitoring and evaluation of rural developmentprograms;- High proficiency in the English language, and very good knowledge oftechnical English concerning the specific sector.General professional experience- Adequate knowledge of the EU rural development policy, programssupported by Structural Funds as well as pre-accession instruments,especially SAPARD Program, for the countries in Central and EasternEurope.Specific professional experience- Minimum 3 years experience in the analysis and evaluation of ruraldevelopment programs in Central and Eastern Europe Countries;- Number of projects on rural development monitoring and evaluation inwhich the expert was involved (minimum 2);- Adequate and substantial knowledge of SAPARD Program functioningand socio-economic situation of rural areas in Central and EasternEuropean Countries, as well as rural development major issues.TERMS OF REFERENCE - Updating of Mid-Term Evaluation of the SAPARD Program 40

Key expert 3 – Environment Impact Assessment, expert II gradeQualifications and skills:- Superior studies in the field of agrarian economics and environment;- Specialization studies in environment management;- High proficiency in the English language, and very good knowledge oftechnical English concerning the specific sector.General professional experience- Adequate knowledge of the EU environmental policies;- Adequate knowledge of the EU pre-accession financial instruments forthe countries in Central and Eastern Europe.Specific professional experience- Minimum 3 years experience in the implementation of EnvironmentImpact Assessments (EIA) of agricultural and rural developmentprograms;- Environmental Impact assessment experience in rural developmentprojects or programs (minimum 2 years);- Number of projects on Environmental Impact Assessment (EIA) of ruraldevelopment plans in which the expert was involved (minimum 2);- Experience in the EIA for programs financed by EU Structural Fundsand/or for SAPARD Program in Central and Eastern EuropeanCountries.6.3.2 Profile of Romanian ExpertsKey expert 4 – Assistant Project Team Leader, expert II gradeQualifications and skills- Economical or technical superior studies in the field of agri-food or ruraldevelopment;- Specialization studies in management and/or monitoring and evaluation;- High proficiency in the English language.General professional experience- Adequate knowledge of the EU pre-accession financial instruments forCentral and Eastern European Countries.Specific professional experienceTERMS OF REFERENCE - Updating of Mid-Term Evaluation of the SAPARD Program 41

- Minimum 2 years experience in projects or programs coordination;- Minimum 3 year experience in the analysis and evaluation of ruraldevelopment programs in Romania and in the implementation oftechnical assistance projects in this field;- Number of projects of monitoring and evaluation for rural developmentprograms in which the expert was involved (minimum 2);- Adequate and substantial knowledge of Romanian National Program forAgriculture and Rural Development 2000-2006.Key expert 5 – Expert in the Evaluation of Rural Development Programs,expert II gradeQualifications and skills- Economical or technical superior studies in the field of agri-food or ruraldevelopment;- Specialization studies in monitoring and evaluation of rural developmentprograms;- High proficiency in the English language.General professional experience- Adequate knowledge of rural development policy and EU pre-accessionfinancial instruments for Central and Eastern European Countries;- Adequate knowledge of the social and economic situation in Romanianrural areas and of major rural development issues.Specific professional experience- Minimum 3 years experience in the analysis and evaluation of ruraldevelopment programs in Romania;- Number of projects on rural development monitoring and evaluationprograms in which the expert was involved (minimum 2);- Adequate and substantial knowledge of Romanian Program forAgriculture and Rural Development 2000-2006.TERMS OF REFERENCE - Updating of Mid-Term Evaluation of the SAPARD Program 42

Key expert 6 – Agrarian/Rural development economist, expert II gradeQualifications and skills- Economical or technical superior studies in the field of agri-food or ruraldevelopment;- Specialization studies in agriculture and rural development economics;- High proficiency in the English language.General professional experience- Adequate knowledge of EU agricultural and rural development policiesand of pre-accession financial instruments.Specific professional experience- Minimum 3 years of professional experience and adequate knowledge ofagricultural and rural economy of Romania;- Number of projects on monitoring and evaluation of agriculture and ruraldevelopment in which the expert was involved (minimum 2).Key expert 7 – Environment Impact Assessment, expert II gradeQualifications and skills:- Superior studies in the field of agrarian economics and environment;- Specialization studies in environment management;- High proficiency in English language.General professional experience- Adequate knowledge of the EU environmental policy and pre-accessionfinancial instruments for Central and Eastern European Countries;- Adequate knowledge of SAPARD Program in Romania.Specific professional experience- Minimum 3 years experience in the implementation of EnvironmentImpact Assessments (EIA) of agricultural and rural developmentprograms;TERMS OF REFERENCE - Updating of Mid-Term Evaluation of the SAPARD Program 43

- Impact assessment experience in rural development project or programs(minimum 2 years);- Number of projects on Environmental Impact Assessment (EIA) of ruraldevelopment programs, in which the expert was involved (minimum 2).Key expert 8 – Technical translator, expert III gradeQualifications and skills- Degree in foreign languages, specialising in English;- High proficiency in technical English.General professional experience- Adequate knowledge of project cycle management terminology, as wellas European Union policies.Specific professional experience- Minimum 3 years of professional experience, preferably in internationalagricultural and rural economy projects (including investments, ecology,etc.);- Number of projects in which the expert was involved (minimum 2);The Contractor could select and employ other experts according to thespecific contract activities. He will ensure the effective participation, wherepossible, of autochthonous specialised staff and suitable autochthonous orinternational staff within the project teams. Experts should not be in interestconflict as regards their responsibilities under the contract.7. REPORTSDuring the activities aimed at producing the first version, the draft finalversion and the Final Mid-Term Evaluation Updating Report (see point 7.3),the evaluation Team will also prepare three reports to inform theContracting Authority of the work progress, the results achieved and theproblems eventually encountered (see points 7.1 and 7.2).TERMS OF REFERENCE - Updating of Mid-Term Evaluation of the SAPARD Program 44

7.1 Inception ReportAn Inception Report shall be submitted to the Contracting Authority forapproval and discussion within 15 days after the contract start-up. TheInception Report will set out a detailed work plan, analysing the dataavailability situation and presenting the data collection and analysismethodology.The contractor should carry out a need assessment and, together with thebeneficiary, prepare the technical specifications of the equipmentnecessary to be supplied.7.2 Activity ReportsThe Evaluator shall present to the Contracting Authority one interim activityreport and one final activity report. These reports shall describe theundertaken activities, results achieved and problems faced in the collectionof reliable data so that due measures can be adopted to solve theproblems. The reports shall also contain detailed information on theorganized meetings, visits and trips.The interim and the final reports shall be presented to the ContractingAuthority, respectively at the end of the third and sixth month of activity.All reports, in Romanian and English languages, signed by Contractor,should be submitted to MAFRD-Managing Authority in four paper copies,two in English and two in Romanian, and two on electronic support (CD-ROM, 3½ Floppy Disk), one in English and one in Romanian.7.3 Mid-Term Evaluation Updating ReportThe first version of the Mid-Term Evaluation Update Report of the SAPARDProgram implementation in Romania should be submitted by the Evaluatorto the Managing Authority no later than 10 th of October 2005, for the firstdiscussion with the Contracting Authority.The draft final version of the Mid-Term Evaluation Update Report should besubmitted by the Evaluator to the Contracting Authority no later than 10 th ofNovember 2005, so that enough time should be allocated for examinationby the Monitoring Committee and possible adjustment or completion of theReport.TERMS OF REFERENCE - Updating of Mid-Term Evaluation of the SAPARD Program 45

The Final Mid-Term Evaluation Updating Report must be submitted to theContracting Authority no later than 10 th of December 2005.In case the Managing Authority or the Monitoring Committee require orrecommend modifications to the reports mentioned in the previousparagraphs, the evaluator must incorporate them and resubmit therespective reports in no more than 5 days since the request.The first version, the draft final version and the Final Mid-Term EvaluationUpdate Report of the SAPARD Program implementation in Romania mustrespect the structure described at point 4.3.2.9 of these ToR and should besubmitted in 6 hard copies, 3 in English and 3 in Romanian, and four onelectronic support (CD-ROM, 3½ Floppy Disk), 2 in English and 2 inRomanian.The Management Authority, within MAFRD, is responsible for theimplementation of the project and should ensure the supervision and thenecessary monitoring.Telephone: 004-021-3072442 or 004-021-3078565Fax: 004-021-3078606E-mail: valentin.toma@maa.roContact person: Mr. Valentin Toma, Counselor8. MONITORING AND EVALUATION8.1 Evaluation criteriaThe assessment of the mid-term evaluation update report shall be basedon the ability of the evaluator to meet the requirements for evaluation reportcontents and evaluation process set in the Commission guidelines:"Guidelines for the evaluation of Rural Development Programs supportedby SAPARD" and "Guidelines for the mid-term evaluation of ruraldevelopment Programs funded by SAPARD".The quality of the evaluation shall depend on the ability of the evaluator toprovide pertinent and credible feedback from final beneficiaries, regionaldirectorates and internal audit unit of the SAPARD Agency, professionalassociations and non governmental organizations, local authorities anddevelopment agencies.TERMS OF REFERENCE - Updating of Mid-Term Evaluation of the SAPARD Program 46

The following criteria for assessing its quality should be applied based onthe Guidelines of the European Commission (MEANS, Vol.1, page 179):-Meeting the needs: Does the evaluation correspond to information needsexpressed by the Terms of Reference requirements and does it answer inan appropriate way to the common evaluation questions?-Methodology adequacy: is the analysis method consistent and sufficientlystrong to provide requested answer to the evaluation questions?-Relevant scope: Have the relevant element been fully examined(principles and actions within the Program, outputs, results and their impactincluding the inter-action between them as well as the consequences of thepolicies established or unpredicted)?-Defensible design: Is the approach of evaluation adequate and appropriateso that the entire set of information including methodological limitationscould answer the evaluation questions?-Reliable data: To what extent the collected, elaborated and selectedprimary and secondary data are appropriate and offer a high degree ofreliability?-Sound analysis: Have the quantitative and qualitative information beenstudied properly and systematically so that correct answers should beprovided to the evaluation questions?-Credible results: Are the results logical and justified by the analysis basedon data collection and are they properly presented?-Impartial conclusions: Are the recommendations correct, uninfluenced byopinions from personnel or interests holders and are they sufficientlydetailed to be operational?-Clear Report: Is the Report describing clearly the evaluated Programincluding its framework and scope together with the procedures and resultsof evaluation so that the information can be easily understood?- Usefulness of the recommendations: do recommendations logically arisefrom the evaluations’ findings and conclusions? Are recommendationsuseful to the improvement of the final phase of the SAPARD Programimplementation and to the following rural development programmingperiod?TERMS OF REFERENCE - Updating of Mid-Term Evaluation of the SAPARD Program 47

8.2 Special requirementsInformation and documents regarding the results and other additionaldocuments, used, received or elaborated by the Contractor during theproject implementation, linked to the SAPARD Program and actiondeveloped, should be classified with a high degree of confidentiality andshould not be used by Contractor without the express agreement of theMAFRD-Managing Authority and the SAPARD Agency.The contractor should not issue any public declaration, or issue or presentany document concerning MAFRD-Managing Authority and SAPARDAgency, on theirs own name or without previous agreement.TERMS OF REFERENCE - Updating of Mid-Term Evaluation of the SAPARD Program 48