program evaluation

Briefing on 

PERFORMANCE 

MANAGEMENT 

and 

PROGRAM EVALUATION 

6 June 2008 

KOSHY THOMAS 

bahagian analisa cukai 

KOSHY THOMAS MoF MALAYSIA

presentation will cover 

• performance management 

• modified budgeting system 

• program evaluation 

• internalised self evaluation 

• jawatankuasa pemandu penilaian 

program








performance management 

• performance management is a 

process of defining a mission and 

desired outcomes, setting 

performance standards, linking 

budgets to performance, reporting 

results, and holding public officials 

accountable for those results 

…….National Academy of Public Administration 

United Kingdom 


performance measurement 

• is the comparison of actual levels of 

performance to pre-established 

target levels of performance 

• to be effective, performance 

measurement must be linked to the 

organizational strategic plan 


how we measure performance 

will depend largely on: 

~ the strategic framework 

~ the strategic plan 

~ the targets 

~ the M&E framework, and 

~ the reporting mechanism 


strategic program cycle 

planning 

evaluation 

RBM 

BSC 

VBM 

EFQM 

implementation 

monitoring 









Chart 2: PROGRAM AGREEMENT & ProLL RELATIONSHIP 

T 

I 

M 

E 

L 

I 

N 

E 

ProLL CHART 

Beneficiaries 

Clients 

Needs/Problems 

Policies 

Activities 

Input 

Process 

Task 

Outputs 

Impact/Outcome 

© Rasappan 1996 

Need 

Fulfillment 

Long Term 

Strategies 

Short Term 

Strategies 

Objectives 

EVALUATION 

NEEDS 

APPRAISAL/ 

EX ANTE 

APPROPRIATENESS/ 

EFFECTIVENESS 

EFFICIENCY 

IMPACT/ 

EX POST 

PROGRAM AGREEMENT 

CLIENT IDETIFICATION 

• Client (Internal, External, Direct, Indirect) 

• Stakeholders (Direct Indirect) 

• Target Groups 

• Service Providers (Internal & External) 

NEEDS/PROBLEM ASSESSMENT 

• Identify Real Needs & Problems 

• Associate it with Clients Groups 

• How to Address These Needs 

Problems 

POLICIES 

• Activity Mandates (Legal/Administrative) 

• Sectoral & National Policies 

ACTIVITIES & OBJECTIVES 

• Identification of Functions 

• To Achieve Objectives 

• To fulfill Needs & Overcome Problems 

INPUTS & STRATEGIES 

• Seek Money, Man & Methods by Function 

• To Implement Activities by Function 

• Through Specific Strategies 

TASK 

• Undertaken by Man using Money and 

Methods 

• Resource Optimisation key to Success 

OUTPUTS 

• Specification of Goods and Services in 

• Qnty, Qlty, Time & Cost 

IMPACT INDICATORS/EVALUATION 

• Key Performance indicators 

• Key Result Areas 

• Proposed Evaluation Plan 

OBJECTIVE 

ACHIEVED 


aligning strategy 

• national level-vision plan, OPP 

• sector level-program goals 

• ministry level-agency goals,AMB1 

• activity level-activity goals, ABM2 


ABM 1 

agency level information 

• agency objectives 

• source of authority of the agency 

• agency strategies 

• agency financial resources (aggregated) 

• agency manpower resources (aggregated) 


PROGRAM ACTIVITY STRUCTURE 

ACTIVITY 1 

DESIGNS 

SUB ACT.1 

ICT/ADMN FUNC 

FUNCTION 1 

Hill Terrain 

TASK 1 

Taskforce Meetings 

SUB ACT.2 

CORE BUSINESS 

Road Designs 

FUNCTION 2 

Wet Lands/Swamp 

TASK 2 

Design Preparations 

SUB ACT.3 

CORE BUSINESS 

Bridge Designs 

FUNCTION 3 

Coastal 

TASK 3 

Data & Research 

All Aspects 

of the 

Activity 

is 

Documented 

in the 

Program 

Agreement 

MAN, MONEY AND METHODS EMPLOYED IN GENERATING THE 

DESIRED OUTPUTS AT THE ACTIVITY LEVEL 









FEATURES OF MBS 

EXPENDITURE TARGET 

PROGRAM AGREEMENT 

AND EXCEPTIONS REPORT 


A MORE GENERALIZED 

APPROACH TO 

EXPENDITURE CONTROL 

Perbendaharaan Malaysia 


what is evaluation? 

a systematic approach to asking 

questions about what a program is doing 

and providing evidence of what it is 

accomplishing in order to provide 

guidance for future direction 


evaluation assists agencies to: 

• better manage its programs 

• improve the delivery of its programs 

• challenge the direction of its program 

• allow the organization to account for its 

performance 


utility of program evaluation 

• to improve program performance 

• increase accountability 

• improve quality of management decisions 

• improve distribution of resources 

• inter agency performance can be 

measured 

• management tool to improve program 

performance 


formative evaluation 

• a continuous or on going assessment 

of the efficiency and effectiveness of a 

program’s processes 

• to ensure that the program is on track 

towards achieving the mandated 

outcomes 

• concurrent or predictive in nature 

(preventive) 


summative evaluation 

• a critical assessment of the performance of 

a program to determine its effectiveness, 

efficiency, economy and appropriateness to 

assist decision makers decide the need for, 

nature of & continued existence of prevailing 

programs. 


evaluation issues & focus 

• appropriateness 

• effectiveness 

• efficiency 

• economy 


APPROPRIATENESS EVALUATION 

assist decision makers in deciding whether 

there is a need for a new program and if so, 

should the government deliver it; 

OR 

whether there is a continuing need for an 

existing program, and if so, should the 

government deliver it. 

© ARTD Malaysia 


EFFECTIVENESS EVALUATIONS 

concerned with measuring the extent to 

which objectives and outcomes are 

being achieved, measuring the factors 

that affect achievement of the outcomes, 

AND 

establishing cause-effect interpretations 

about the extent to which it can be 

claimed that the program caused the 

outcomes. 



EFFICIENCY EVALUATION 

examines the extent to which 

program inputs are minimized for a 

given level of program outputs 



ECONOMIC EVALUATION 

the term economy can often be used 

to distinguish between being 

efficient and yet being wasteful 










what is internal evaluation? 

• internal evaluation is evaluation conducted 

by a person or group directly associated 

with the organization or program that 

supports organisational development and 

intentional change 

• under the new paradigm, development 

evaluation is being viewed as an ongoing 

learning process 

KOSHY THOMAS MoF MALAYSIA 

© ARTD Malaysia

major internal evaluation design 

strategies 

• external contractors 

– evaluation, accountability, comparative learning 

– not permanent employees 

– independent of program teams and managers 

– report and recommend 

• independent internal evaluation unit 

– evaluation and continuous quality improvement 

(CQI) 

– may include planning, accountability, learning 

– independent of program teams and managers 

– report, recommend, advocate 


Adapted from: Love (1998)

major internal evaluation design 

strategies 

• embedded internal evaluation function 

– evaluative inquiry, monitoring, CQI, OD, learning 

– responsibility of managers or team leaders 

– integral part of program and project teams 

– report, recommend, advocate, manage, fix 

• hybrid internal evaluation function 

– evaluation, CQI, accountability, OD, learning 

– external contract evaluators link with internal 

evaluation unit or staff with evaluation roles 

– report, recommend, advocate, manage, fix, plan 


Adapted from: Love (1998)

IE MODEL 

• have ownership over the evaluation process. 

• objective to improve not to prove. 

• facilitate using the process and results for program improvement 

and decision making. 

• improved communication and shared understanding between 

different groups 

• evaluation is ingrained in the day-to-day operations of an 

organisation 

• evaluation is a developmental process, not only a report card 

process 

• an important benefit of evaluation is organisational learning -- a 

way for the organisation to assess its progress and to change in 

ways that lead to greater effectiveness 


disadvantages of internal 

evaluation 

• less perceived credibility than an external 

evaluator 

• objectivity may be questioned 

• greater potential for bias 

• potential for role confusion 

• workload may delay completion of evaluations 


program evaluation 

implementation structure 


EVALUATION COMMITTEE 

(Treasury) 

STEERING COMMITTEE 

(Agency) 

STUDY COMMITTEE/TEAM 

(Activity) 


COORDINATING OFFICER 

(Treasury) 

FACILITATOR 

(Agency) 

TEAM LEADER 

(Activity) 


EVALUATION FLOWCHART

ProLL Evaluation Flowchart 

© 1999 Arunaselam Rasappan/Jerry Winston 

Public Sector Policy & Program Environment 

Doing The Right Things 

The Right Way 

For The Right People 

Logical Linkages with Substantive Purpose of Program 

Resource Program 

Efficacy Improvement Compliance 

Design Adjustment 

Data Collection 

Facilitate 

Decision-making 

Financial Accountability 

Management Accountability 

Program Accountability 

MBS Program Agreement: 

Pilot Testing 

• Data Collection 

• Data Tabulation & Analysis 

• Data Interpretation 

• Validity & Reliability 

Testing 

Data Entry 

Stakeholder Utilization 

Verify Client Needs & Objectives 

Evaluability Assessment 

Determine Evaluation Questions 

• Identify pre-determined eval 

issues 

• Analyse program agreement 

• Conduct stakeholder inventory 

• Analyse program logic 

Prioritze Evaluation 

Questions 

Data Sources and 

Instrumentation 

Determine Research Design 

Determine Evaluation Design 

Information Needed for 

Evaluation 

Data Validation 

Data Analysis 

Findings & 

Conclusions 

• Stakeholder Analysis 

of Report 

• Stakeholder 

Feedback 

Evaluation Reporting 

• Multiple reports 

• Multiple audiences 

• User-oriented reports 

• Utilization-focussed 

reports 

Preliminary Planning & 

Assessment Stage 

Evaluation Design 

Stage 

Data Collection & 

Analysis Stage 

Reporting & 

Information 

35 

Utilization Stage

ISE process

FLOW CHART FOR PUBLIC SECTOR PROGRAM EVALUATION UNDER THE THE MODIFIED BUDGETING SYSTEM (MBS) (MBS) 


STEERING COMMITTEE 

WORKSHOP 11 

PLANNING 

WORKSHOP 2 2 

INSTRUMENT/CONSTRUCT 

WORKSHOP 3 3 

DATA DATA ANALYSIS/REPORT WRITING WRITING 

1 

BY 

APPROVAL BY 

CONTROLLING OFFICER 

1 CONTROLLING OFFICER 

4 

4 

INTERNAL 

ASSESSMENT 

INTERNAL 

ASSESSMENT 

2 

2 

PARTICIPATION 

ANALYSIS 

PARTICIPATION 

ANALYSIS 

5 

5 

PROGRAM LOGIC 

PROGRAM LOGIC 

1 

2 

1 

2 

No 

No 

Yes 

Yes 

No 

No 

Yes 

Yes 

Yes 

29 

3 

3 

No 

No 

11 

11 

Yes 

29 

17 

17 

18 

18 

19 

19 

20 

20 

27 

27 

28 

28 

EVALUABILITY 

ASSESMENT 

EVALUABILITY 

ASSESMENT 

STEERING COM 

ACTION PLAN 

STEERING COM 

ACTION PLAN 

STEERING COM 

INSTRUMENT 

STEERING COM 

INSTRUMENT 

DATA 

COLLECTION 

DATA 

COLLECTION 

DATA ENTRY 

DATA ENTRY 

DATA CLEANING 

DATA CLEANING 

STEERING COM 

DRAFT REPORT 

STEERING COM 

DRAFT REPORT 

FINAL REPORT 

FINAL REPORT 

FOLLOW-UP 

ACTION 

FOLLOW-UP 

ACTION 

1 

1 

6 

6 

7 

7 

8 

8 

9 

9 

10 

10 

KRA/KPI/CSF 

KRA/KPI/CSF 

OUTCOMES 

HIERARCHY 

OUTCOMES 

HIERARCHY 

STAKEHOLDER 

ANALYSIS 

STAKEHOLDER 

ANALYSIS 

EVALUATION 

QUESTION 

EVALUATION 

QUESTION 

ACTION PLAN 

ACTION PLAN 

Cross Referencing 

Cross Referencing 

2 Run Sequence Again 

2 

Run Sequence Again 

1 

1 

ISOLATE M’ABLE 

INDICATORS 

ISOLATE M’ABLE 

INDICATORS 

IDENTIFY DATA 

SOUURCES 

IDENTIFY DATA 

SOUURCES 

DRAFT 

INSTRUMENT 

DRAFT 

INSTRUMENT 

FIELD TEST/ 

VALIDATION 

FIELD TEST/ 

VALIDATION 

SAMPLING DESIGN 

12 

13 

14 

15 

16 

12 

13 

14 

15 

SAMPLING DESIGN 16 

2 

2 

RUN BASIC STATS 

RUN BASIC STATS 

IDENTIY CROSS-TABS 

EVALUATION Q. 

GROUPING 

QUESTIONAIRE 

ITEM ANALYSIS 

CONCLUSION & 

RECOMMENDATION 

DRAFT FIRST 

REPORT 

21 

22 

IDENTIY CROSS-TABS22 

EVALUATION Q. 

GROUPING 

QUESTIONAIRE 

ITEM ANALYSIS 

23 

24 

25 

CONCLUSION & 25 

RECOMMENDATION 

DRAFT FIRST 

REPORT 

26 

21 

23 

24 

26 

© KOSHY THOMAS MoF, Malaysia 2004

controls mechanisms 

• peer review through the steering 

committee 

• meta evaluation 

• internal audits

POLICY MAKERS AND 

PLANNERS SHOULD KNOW: 

• whether the program is producing the 

desired outcomes. 

• whether there is still a continued need for 

the program, if so, what are its priorities. 

• how efficiently is the program producing the 

desired results. 

• what are the alternatives available in 

delivering the program. 

• whether the program should be adjusted, 

modified or terminated. 


PROGRAM MANAGERS SHOULD 

KNOW: 

‣ how and why the program works and in what 

circumstances. 

‣ goal to improve the program delivery. 

‣ framework strategies for continuous improvement. 

‣ program effectiveness and continued funding. 

‣ discover new knowledge about the effective 

practice. 

‣ improved communication and shared 

understanding between different groups. 

‣ evaluation becomes part of program intervention 

rather then ‘stand alone’ initiative 


terima kasih 

thank you 

koshy@treasury.gov.my

program evaluation

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?