Annex: Methodology 1 STRATEGY - MADR

Updating of Mid-Term Evaluation of SAPARD in Romania for the period 2000-2005and once the impact has been established,• relevance: is the initiative still viable in relation to the sector priorities, or not?• sustainability: will the positive effects of the initiative continue once it has beenconcluded?Assessment of the effectiveness and of the efficiency, in particular, is necessary at thevarious levels of performance of the programme:the assessment of effectiveness will regard the relationships:• Global Objectives/Global Impact Indicators;• Specific Objectives / Indicators of Result;• Operative Objectives / Indicators of Output.the assessment of efficiency will regard the relationships:• Financial Plan / Financial Indicators;• Specific Objectives / Unit Costs.Figure n° 3 has helped us to distinguish the two components of effectiveness and efficiency.In the right part of the figure the elements necessary for assessment of effectiveness areidentified, and in the left part those for assessment of efficiency.With respect to the impact assessment, account has been taken of the difficulties linked withestimating the added value due to the programme measures. Hence, we have taken into dueconsideration the occurrence of factors exogenous to the generation of the impacts; thegross impacts of the measures have been distinguished from the net impacts directlyimputable to the implementation of the measures, where possible. To obtain the net effect ithas been necessary to take into adequate account three types of “disturbance factors”, whichare deducted from the gross effect:• deadweight effect: this is an effect that would have taken place in any case, evenin the absence of the programme;• substitution effect: this occurs when a change in the situation of an operator or ofa group of operators who are direct beneficiaries of the support of the programmeis obtained to the detriment of operators who are not beneficiaries of this support;• displacement effect: this occurs when a change in favour of an operator isaccompanied by undesired effects for operators not participating in theprogramme in the same geographical area of intervention.6

Updating of Mid-Term Evaluation of SAPARD in Romania for the period 2000-2005Figure n° 3 - Assessment of effectiveness and efficiencyGLOBAL OBJECTIVES(PRIORITY AXIS)34FINANCIAL PLAN5SPECIFIC OBJECTIVES2OPERATIVE OBJECTIVES(PROJECTS TYPOLOGIES)1IMPLEMENTATION MODALITIESFINANCIALINDICATORSCOMMITMENTS& PAYMENTSIMPLEMENTATIONOF PROJECTSOUTPUTINDICATORSUNIT COSTSINDICATOR OFRESULTSSPECIFICIMPACTSSPECIFICIMPACTINDICATORSGLOBALIMPACTSGLOBAL IMPACTINDICATORS123Assessment of effectiveness45Assessment of efficiency7

Updating of Mid-Term Evaluation of SAPARD in Romania for the period 2000-20051.3 Methodological ApproachMaking reference to the project objectives we describe in greater detail the four key elements of theevaluation process, as listed below:‣ Common Evaluation Questions (CEQ)‣ Indicators‣ Sources of data‣ Modalities of transfer2.3.1 The Common Evaluation QuestionsThe Common Evaluation Questions 1 (CEQ) are the instrument of reference that is used by theEvaluator in the course of the activities to provide the EC with performance information.The CEQ is a complex instrument which, also due to its aims of re-aggregating results atCommunity level, has a rigid structure and is accompanied by guidelines of analytical anddetailed use.The Common Evaluation Questions for SAPARD are organised into three chapters:• Chapter A - Measure-specific questions: these are the majority of the questions. Theyconcern effects arising from the activities within each of the fifteen measures ofRegulation 1268/99;• Chapter B – Cross-cutting questions: regarding the overall impacts of SAPARD in termsof results and implementation. They provide a summary concerning the main effects ofthe programme in relation to the means and objectives of Regulation 1268/99.• Programme – specific questions with criteria and indicators for each individualprogramme to be filled by the applicant countries.• Chapter C – Global assessment questions on the overall affects of SAPARD. Theyshould be addressed on the basis of responses to the two previous chapters.In all, the questionnaire contains 57 questions, which the evaluation has answered, where possibleand relevant, setting out as many as 98 evaluation criteria and 129 indicators.In the case of the SAPARD Programme in Romania, there are six accredited measures, asevidenced in the following table n° 2, with which 27 questions, 37 criteria and 48 indicators areassociated. In particular, Measures 1.1, 2.1 and 4.2 were accredited on 31st of July 2002 whileMeasures 3.1, 3.4 and 4.1 on 5th of December 2003. Other measures contained in theNational Program for Agriculture and Rural Development (NPARD) has been accredited in2005, others have been cancelled.The evaluation activities have included an analysis of the measures whose implementation hasbeen started by September 2005 and an analysis of the measures whose implementation will bestarted from December 2005, but which are to be achieved in accordance with the requirementsand guidelines provided for the ex-ante evaluation of the SAPARD Programme.1 Cf. Guidelines for the Evaluation of Rural Development Programmes supported by SAPARD, Section D. April2001.8

Updating of Mid-Term Evaluation of SAPARD in Romania for the period 2000-2005Table n° 2 - Common Evaluation Questions – Questions, Criteria, IndicatorsCommon Evaluation Questions Questions Criteria IndicatorsChapter A: Measure - specific questionsII1.1Improvement of processing and marketing ofthe agricultural and fishery products7 11 16XII2.1Development and improvement of ruralinfrastructure3 5 5I 3.1 Investments in agricultural holdings 7 8 12V3.4Development and diversification of economicactivities, multiple activities, alternative income6 7 6XI 4.1 Improvement of the vocational training 2 3 4XV 4.2 Technical assistance 2 3 5Total Accredited measures 27 37 48III 1.2Improving the structures for quality control, forthe quality of foodstuffs and for consumerprotection2 5 5XIII 2.2 Management of water resources (cancelled) 3 7 11VII 3.2 Setting-up producer groups 3 5 6IV3.3Agricultural production methods designed toprotect the environment and maintain thecountryside3 4 6XIV 3.5 Forestry 5 11 14Chapter B: Cross – cuttingB.I.B.II.Concerning the objective: to contribute to theimplementation of the acquis communautaireconcerning the CAPConcerning the objective: to solve priority andspecific problems for the sustainableadaptation of the agricultural sector and ruralareas in the applicant countries2 12 145 10 12B.III.Concerning the conception and implementationof the programme1 1 3Programme specific Questions 6 6 10Total 57 98 129The Questionnaire, apart from the specific questions for each measure, contains a number ofquestions applicable to the programme as a whole, regarding the overall impact of SAPARD interms of results and implementation.It has been the task to integrate the CEQ with the specific evaluation questions if its structureshould not be considered suitable to represent the situation adequately.Furthermore, we have developed a set of EQ related to the evaluation of the administrative setupof the programme. The requirement to incorporate an evaluation of the administrative set-upis integrated in the Midterm specific guidelines at a late phase, meaning that the Commissionhas not developed specific evaluation questions for this purpose.9

Updating of Mid-Term Evaluation of SAPARD in Romania for the period 2000-20051.3.1.1 Methods of compilation of the Common Evaluation QuestionsThe objective of this paragraph is to set out the methods used of compiling the evaluationquestionnaire; in answering the single questions all pertinent information (including the probablecause/effect relations, the influence of the context, data intersection, etc.) analysed andjudgements formulated in the form of answers to the common questions and to the other elementsof evaluation.Starting from each question, the Evaluator therefore has identified the elements that couldinfluence the analysis and in particular we have proceeded with the:• construction of the “intervention logic” to identify the elements and to describe theirinteraction in reaching the objective foreseen by the question;• identification of the evaluation criteria, which supply the elements of judgement on thebasis whereof to analyse the question under examination;• identification of the indicators through which to provide the qualitative/quantitativeanswers for each evaluation criterion.A series of criteria and of “common” indicators are defined in the CEQ, which we have used inanswering the various questions. Moreover, in addition to those foreseen in the CEQ attached withthe Terms of Reference, specific questions, criteria and indicators have been defined whendrawing up the evaluation conditions.Apart from the above elements (logic of intervention, evaluation criteria and indicators) theCommission wants identified a further element of analysis, represented by the target level. Thetarget level represents the level that an indicator should reach to satisfy the correspondingevaluation criterion. The definition of this level should have been made at the time ofprogramming the measures and should logically correspond to the objectives quantified in theprogramme. In the course of our activities, we have verified the presence of this value in theprogramme.In order to give adequate and satisfactory answers to the evaluation questions and, therefore,to obtain one or more sufficiently robust estimates of the effects of the measures, we haveused as far as possible existing information (i.e. secondary data). These concernadministrative data, documentation pertaining to the management of the programme, officialstatistics, studies and researches already carried out, as well as information relating to similarmeasures, concerning the magnitude of the population’s needs and the importance of thedisparities to be reduced; appreciate weakness or capacities and whether the programmeobjectives are pertinent with respect to a broader context.As opposed to operating data (programme monitoring), which concentrate on the operators and onthe direct beneficiaries of the aid, the context indicators refer to the whole population or territory(whose characteristics can be studied having historical series available) quite apart from theirinvolvement in the programme. Comparison with a statistical estimate before and afterwardscannot indicate anything regarding the estimate of an impact. However it supplies informationusable within the framework of an impact analysis, whose role is that of imputing the changeobserved to a number of causes: the programme on the one hand and the exogenous factors onthe other.The advantage of this methodology consists in the possibility of identifying the contribution ofeach measure towards achieving the overall plan objectives. This evaluation approach has10

Updating of Mid-Term Evaluation of SAPARD in Romania for the period 2000-2005established the net effect of the plan, and for this purpose the Evaluator has decided what typeof comparison to make among the following, according to the availability of information:‣ temporal: this implies comparing the situation before/after for the beneficiaries.Information is required on the initial situation, i.e. the level of the indicator before theparticipation of an individual or of an agency in the development plan. This informationshould normally be collected in the initial programming phases. The initial situation cansometimes be deduced from the evaluation (using for example data collected for theanalysis). The results obtained are, however, susceptible to exogenous factors, so thatit is difficult to isolate the net effect;‣ counterfactual (“policy-off”): this entails a comparison with the situation that would havebeen the case without the measure. Information is required also on the beneficiaries (forexample by sampling). This comparison can eliminate the influence of exogenousfactors and therefore make it possible to establish the net effect of the plan;‣ normative: this means identifying good practices and benchmarking; information isrequired on a given norm, on the best practice obtained earlier or in other regions, etc.1.3.2 Indicators and relative sources of informationThe existence of a system of indicators is definitely one of the elements that most influences theavailability of a programme. The indicators are in fact the main instruments through which themonitoring information is obtained, thus enabling the progress of a programme to be verified andassessed, and the elements necessary for the evaluation of the results and of the impacts derivingfrom the investments funded through the programme to be supplied. Monitoring and evaluationform part and parcel of the programming cycle and are closely linked together, as shown in thefollowing figure:Figure n° 4 -Monitoring and evaluation in the programme cycleEX-ANTEEVALUATIONSupport toFORMULATION OFPROGRAMMES & PLANSMONITORING:• Financial• Physical• Of results• Of Impact• Of ProceduresMID-TERMEVALUATIONAnd its UpdateCollection/definitionInformation/IndicatorsAnalysisof resultsIMPLEMENTATION OF PROGRAMMES &PLANS:• Definition of Procedures• Information and Sensitization• Selection of Projects• Implementation of ProjectsMID-TERM REVISIONS ANDMODIFICATIONSEX-POSTEVALUATION11

Updating of Mid-Term Evaluation of SAPARD in Romania for the period 2000-2005In making a reconnaissance of the system of indicators available and their possible integration,account has been taken of certain criteria, such as:• capacity to reflect adequately the state of the outputs;• clarity of definition;• adherence to the questions posed by the CEQ.A first distinction is made between the programme indicators and the context indicators. Theformer aim representing the direct effects of the programme on the portion of territory orpopulation concerned by it, while the latter refers to the entire territory and/or to the populationas a whole, enabling the results of the actions to be contextualised.The output indicators are:‣ financial indicators whereby the level of mobilisation of the financial resources ismeasured;‣ physical indicators with which the work progress is measured.Financial Indicators.These are constructed from the ratios between commitments and allocations, distinguishing:• effective capacity of progress, given by the ratio between commitments and allocations;• effective capacity of utilisation, given by the ratio between payments and commitments.Considering also the measures identified and those in process of selection, it has beenpossible to elaborate two other typologies of indicators, that is:• potential capacity to make progress;• potential commitment capacity.Physical IndicatorsTheir definition in integration of those available is linked to the quantification of the physicalobjectives of implementation, so that the distinction is drawn between physical objectivesachieved, achievable and to be achieved.The basic data collected are represented by• physical objectives achieved, referred to the allocation attributed to projects concluded;• physical objectives achievable, referred to the allocation attributed to projects beingimplemented and identified;• physical objectives to be achieved, referred to the allocations attributed to projects inthe selection phase and not yet identified.The use of indicators of result enables the direct results obtained by implementing theactions to be measured.In the course of the mid-term evaluation the indicators of result have maked it possible on theone hand to follow the physical and financial progression of the actions, and on the other handto formulate the first projections as to the achieving of permanent effects. The typology of theindicators of result selected depends on a number of characteristics both of the beneficiaries ofthe measures and of users of the outputs, such as:12

Updating of Mid-Term Evaluation of SAPARD in Romania for the period 2000-2005• characteristics of the beneficiaries (location, average dimensions, etc.);• reaction of the potential beneficiaries (effective beneficiary enterprises of the totalpotential beneficiary enterprises);• capacity of qualitative improvement of supply;• decisions consequent upon the measures (increase in jobs).The Impact Indicators measure the specific impact, consequent upon the outputs and theresults obtained, and the global impact, consequent upon the specific objectives achieved. Themid-term evaluations update focus on the specific impacts; as it is the task of the ex-postevaluation to pay more attention to the global impact.The capacity of the monitoring service to provide the information necessary to support it andconsequently to complete the finalization of the system to collect the primary data, is assed.1.3.3 Sources of dataA first source of information is represented by the monitoring results, by earlier investigationsand studies, and by the information contained in the programming documents. In particular, theaforesaid source has been suitable for providing data of procedural and financial type, and theresults obtained during programme implementation. The secondary data are integrated withstatistical information provided by Qs surveys combined with basic information linked to thelocal context and in particular that of the sector.The direct collection by the Evaluator of the primary data has been a central element of theevaluation system as it makes it possible to obtain original information targeted on theperformance of the service whose qualitative level is closely connected with the functionality ofthe collection. A fair part of the primary data comes from the direct beneficiaries and therepresentativeness of the results guarantees its capacity to provide the programme results andimpacts.Below the data surveyed are classified in homogeneous categories according to the nature ofthe datum, survey source and connection with the individual indicators in the various chaptersof the CEQ, so as to clarify the Evaluator’s approach to the construction of the data surveysystem and to the integration of the sources of primary and secondary data.Nature of DatumSurvey sourcesReference toChapters of theCEQData of an economic, financial,structural and productive nature (levelof use of means of production such aswater, fertilisers, pesticides) relating tothe farms and forestry units, and toinformation regarding the “labour”factor within them.Analytical data relating to thequantitative and qualitative availabilityof ground waters and surface waters.Data relating to the impacts on thenatural systems bordering on thecultivated areas and concerned by themeasures, including data relating tostructural and qualitative aspects of• Monitoring Committee;• Direct inquiries on farms (interviews,questionnaires, case studies);• FADN;• National and regional accountingstatistics• Planning documents.• Monitoring Committee;• Direct inquiries on farms (interviews,questionnaires, case studies);• Sample analytical surveys.• Monitoring Committee;• Targeted investigations conductedoutside of farms (sampling, testareas);• National and regional statistics;Chapter AChapter CChapter AChapter AChapter C13

Updating of Mid-Term Evaluation of SAPARD in Romania for the period 2000-2005the landscape • Information obtained from localsources relating to qualitativeparameters of the environment(quantitative information on species).Data relating to the processing ofagricultural/fishery/forestry products:- hygienic conditions,- work safety,- qualitative characteristics of theproduct,- impact on the environment.Data relating to the characteristics ofthe airData relating to the impacts producedby the programme on the main socioeconomicparameters, such as:work, incomes, environment, other.Data relating to the effects of theprogramme on implementation of theacquis communitaire.• Monitoring Committee;• Investigations conducted inprocessing plants (interviews,sampling, single case studies);• National and regional statisticsrelating to the agro-industrial sector;• Monitoring Committee;• Investigations conducted outside offarms (interviews, sampling)• Monitoring Committee;• FADN ;• National and regional accountingstatistics• EUROSTAT.Chapter AChapter CChapter AChapter B:Cross – cuttingThe problem of sources is central for the reliability of the data; below is the range of instruments ofinvestigation used.Some typologies of investigations, such as the questionnaire for farms adhering to the measures,provide a complex of information that is all the more substantial, the more articulate thequestionnaire. This instrument lends itself to collecting both qualitative and quantitative dataforming samples that are substantial enough to guarantee that they well represent the populationunder examination.Interviews, single or group, are a useful instrument for collecting data from the beneficiaries onmanagement aspects of the programme, duly integrating the information obtained by monitoringsystems.Case studies have been used above all for analysing impacts and providing perspectives to thequantitative analysis.1.4 Project ActivitiesThe completed and remaining project activities are summarized in the table below.ActivitiesActivity 1: Mobilisation of key experts & establishmentof project officeActivity 2: Introductory meeting to construe andinterpret the evaluation demandActivity 3: Reconstruction of the Logical Framework ofthe ProgrammeIndicators ofAchievement‣ Arrival of ProjectTeam Leader‣ Adequate logisticarrangements set up‣ Introductory meetingorganised‣ Evaluation demandconstrued andinterpreted‣ Logical Framework ofthe Programmeprepared14

Updating of Mid-Term Evaluation of SAPARD in Romania for the period 2000-2005ActivitiesActivity 4: Analysis of the results of the Ex-Ante andMid-term evaluations and SWOT analysis validityActivity 5: Analysis of data availability and elaborationof a data collection and analysis methodologyActivity 6: Definition, revision and up-dating of outputand impact indicatorsActivity 7: Preparation of the Inception ReportActivity 8: Analysis of the management and monitoringprocedures and those of activating the financial circuitActivity 9: Setting up of programme data base and dataentryActivity 10: Collection and typology of context dataActivity 11: Collection and typology of primary dataActivity 12: Collection and typology of secondary dataActivity 13: Organization and implementation of theseminarActivity 14: Analysis of primary and secondary dataActivity 15: Analysis of the Institutional ImpactIndicators ofAchievement‣ Ex-Ante and Mid-termevaluation resultsanalysed‣ SWOT analysisvalidity verified‣ Data availabilitysituation analysed‣ Data collection andanalysis methodologyelaborated‣ Indicators defined,revised and up-dated‣ Inception Reportprepared andsubmitted to MA andSteering Committeefor comment‣ Inception Reportapproved‣ Management andmonitoring proceduresand those of activatingthe financial circuitanalysed‣ Data base set up.‣ Data entry assured bydata collectors‣ Context data collectedand data entrycompleted‣ Methods, instruments,timing and modalitiesof collection identified‣ Primary data collectedand data entrycompleted‣ Secondary datacollected and dataentry completed‣ Seminar programmeagreed with MA‣ Training materials areprepared translatedand made available‣ Seminar Organizedand implemented‣ Reactionaryquestionnairecompleted at the endof the seminarPrimary and Secondarydata analysed‣ Institutional Impactanalysed15

Updating of Mid-Term Evaluation of SAPARD in Romania for the period 2000-2005ActivitiesActivity 16: Preparation of the Update Mid-termevaluation reportActivity 17: Organisation of final conferenceIndicators ofAchievement‣ Update of the MidtermEvaluationReport prepared andsubmitted to SteeringCommittee and to theNational SAPARDMonitoring Committeefor comment‣ Update of the MidtermEvaluationReport approved andsubmitted to EC‣ Final Conferencecarried out.16