Annex: Methodology 1 STRATEGY - MADR

Updating of Mid-Term Evaluation of SAPARD in Romania for the period 2000-20051.3.1.1 Methods of compilation of the Common Evaluation QuestionsThe objective of this paragraph is to set out the methods used of compiling the evaluationquestionnaire; in answering the single questions all pertinent information (including the probablecause/effect relations, the influence of the context, data intersection, etc.) analysed andjudgements formulated in the form of answers to the common questions and to the other elementsof evaluation.Starting from each question, the Evaluator therefore has identified the elements that couldinfluence the analysis and in particular we have proceeded with the:• construction of the “intervention logic” to identify the elements and to describe theirinteraction in reaching the objective foreseen by the question;• identification of the evaluation criteria, which supply the elements of judgement on thebasis whereof to analyse the question under examination;• identification of the indicators through which to provide the qualitative/quantitativeanswers for each evaluation criterion.A series of criteria and of “common” indicators are defined in the CEQ, which we have used inanswering the various questions. Moreover, in addition to those foreseen in the CEQ attached withthe Terms of Reference, specific questions, criteria and indicators have been defined whendrawing up the evaluation conditions.Apart from the above elements (logic of intervention, evaluation criteria and indicators) theCommission wants identified a further element of analysis, represented by the target level. Thetarget level represents the level that an indicator should reach to satisfy the correspondingevaluation criterion. The definition of this level should have been made at the time ofprogramming the measures and should logically correspond to the objectives quantified in theprogramme. In the course of our activities, we have verified the presence of this value in theprogramme.In order to give adequate and satisfactory answers to the evaluation questions and, therefore,to obtain one or more sufficiently robust estimates of the effects of the measures, we haveused as far as possible existing information (i.e. secondary data). These concernadministrative data, documentation pertaining to the management of the programme, officialstatistics, studies and researches already carried out, as well as information relating to similarmeasures, concerning the magnitude of the population’s needs and the importance of thedisparities to be reduced; appreciate weakness or capacities and whether the programmeobjectives are pertinent with respect to a broader context.As opposed to operating data (programme monitoring), which concentrate on the operators and onthe direct beneficiaries of the aid, the context indicators refer to the whole population or territory(whose characteristics can be studied having historical series available) quite apart from theirinvolvement in the programme. Comparison with a statistical estimate before and afterwardscannot indicate anything regarding the estimate of an impact. However it supplies informationusable within the framework of an impact analysis, whose role is that of imputing the changeobserved to a number of causes: the programme on the one hand and the exogenous factors onthe other.The advantage of this methodology consists in the possibility of identifying the contribution ofeach measure towards achieving the overall plan objectives. This evaluation approach has10