Annex: Methodology 1 STRATEGY - MADR

More documents

Recommendations

Info

Updating of Mid-Term Evaluation of SAPARD in Romania for the period 2000-20051.3.1.1 Methods of compilation of the Common Evaluation QuestionsThe objective of this paragraph is to set out the methods used of compiling the evaluationquestionnaire; in answering the single questions all pertinent information (including the probablecause/effect relations, the influence of the context, data intersection, etc.) analysed andjudgements formulated in the form of answers to the common questions and to the other elementsof evaluation.Starting from each question, the Evaluator therefore has identified the elements that couldinfluence the analysis and in particular we have proceeded with the:• construction of the “intervention logic” to identify the elements and to describe theirinteraction in reaching the objective foreseen by the question;• identification of the evaluation criteria, which supply the elements of judgement on thebasis whereof to analyse the question under examination;• identification of the indicators through which to provide the qualitative/quantitativeanswers for each evaluation criterion.A series of criteria and of “common” indicators are defined in the CEQ, which we have used inanswering the various questions. Moreover, in addition to those foreseen in the CEQ attached withthe Terms of Reference, specific questions, criteria and indicators have been defined whendrawing up the evaluation conditions.Apart from the above elements (logic of intervention, evaluation criteria and indicators) theCommission wants identified a further element of analysis, represented by the target level. Thetarget level represents the level that an indicator should reach to satisfy the correspondingevaluation criterion. The definition of this level should have been made at the time ofprogramming the measures and should logically correspond to the objectives quantified in theprogramme. In the course of our activities, we have verified the presence of this value in theprogramme.In order to give adequate and satisfactory answers to the evaluation questions and, therefore,to obtain one or more sufficiently robust estimates of the effects of the measures, we haveused as far as possible existing information (i.e. secondary data). These concernadministrative data, documentation pertaining to the management of the programme, officialstatistics, studies and researches already carried out, as well as information relating to similarmeasures, concerning the magnitude of the population’s needs and the importance of thedisparities to be reduced; appreciate weakness or capacities and whether the programmeobjectives are pertinent with respect to a broader context.As opposed to operating data (programme monitoring), which concentrate on the operators and onthe direct beneficiaries of the aid, the context indicators refer to the whole population or territory(whose characteristics can be studied having historical series available) quite apart from theirinvolvement in the programme. Comparison with a statistical estimate before and afterwardscannot indicate anything regarding the estimate of an impact. However it supplies informationusable within the framework of an impact analysis, whose role is that of imputing the changeobserved to a number of causes: the programme on the one hand and the exogenous factors onthe other.The advantage of this methodology consists in the possibility of identifying the contribution ofeach measure towards achieving the overall plan objectives. This evaluation approach has10
Updating of Mid-Term Evaluation of SAPARD in Romania for the period 2000-2005established the net effect of the plan, and for this purpose the Evaluator has decided what typeof comparison to make among the following, according to the availability of information:‣ temporal: this implies comparing the situation before/after for the beneficiaries.Information is required on the initial situation, i.e. the level of the indicator before theparticipation of an individual or of an agency in the development plan. This informationshould normally be collected in the initial programming phases. The initial situation cansometimes be deduced from the evaluation (using for example data collected for theanalysis). The results obtained are, however, susceptible to exogenous factors, so thatit is difficult to isolate the net effect;‣ counterfactual (“policy-off”): this entails a comparison with the situation that would havebeen the case without the measure. Information is required also on the beneficiaries (forexample by sampling). This comparison can eliminate the influence of exogenousfactors and therefore make it possible to establish the net effect of the plan;‣ normative: this means identifying good practices and benchmarking; information isrequired on a given norm, on the best practice obtained earlier or in other regions, etc.1.3.2 Indicators and relative sources of informationThe existence of a system of indicators is definitely one of the elements that most influences theavailability of a programme. The indicators are in fact the main instruments through which themonitoring information is obtained, thus enabling the progress of a programme to be verified andassessed, and the elements necessary for the evaluation of the results and of the impacts derivingfrom the investments funded through the programme to be supplied. Monitoring and evaluationform part and parcel of the programming cycle and are closely linked together, as shown in thefollowing figure:Figure n° 4 -Monitoring and evaluation in the programme cycleEX-ANTEEVALUATIONSupport toFORMULATION OFPROGRAMMES & PLANSMONITORING:• Financial• Physical• Of results• Of Impact• Of ProceduresMID-TERMEVALUATIONAnd its UpdateCollection/definitionInformation/IndicatorsAnalysisof resultsIMPLEMENTATION OF PROGRAMMES &PLANS:• Definition of Procedures• Information and Sensitization• Selection of Projects• Implementation of ProjectsMID-TERM REVISIONS ANDMODIFICATIONSEX-POSTEVALUATION11
Page 6 and 7: Updating of Mid-Term Evaluation of
Page 16: Updating of Mid-Term Evaluation of

Annex: Methodology 1 STRATEGY - MADR

Create successful ePaper yourself

Delete template?

Save as template?