Energy efficiency and Demand Side Management Program ... - Eskom

Energy efficiency and Demand Side
Management Program Evaluation
Guideline Proposal
‐‐‐‐A POET Based Measurement and Verification Approach from the
Engineering, Environmental, Social, and Economic Aspects
CONFIDENTIAL
Prepared by the Centre of New Energy Systems, University of Pretoria
Based on an early version of
ESKOM Energy Audit’s Program Evaluation Guideline Proposal
Updated on 2011-06-23

Contents
1 Introduction .....................................................................................................................................4
2 International EEDSM Evaluation—A Measurement and Verification Approach .............................5
3 The South African Context ...............................................................................................................6
4 EEDSM Program Evaluation Guideline .............................................................................................6
4.1 Contents of Program Evaluation .............................................................................................6
4.2 Purpose ...................................................................................................................................7
4.3 Benefits ...................................................................................................................................8
5 Scope of Work (SOW) .......................................................................................................................8
5.1 Objective .................................................................................................................................9
5.1.1 POET Efficiency ...................................................................................................................9
6 EEDSM Program Evaluation Criteria and Methodology ................................................................ 11
6.1 Evaluation Marking System Development........................................................................... 12
6.1.1 Engineering Aspects ........................................................................................................ 12
6.1.2 Environmental Aspects .................................................................................................... 13
6.1.3 Social Aspects .................................................................................................................. 14
6.1.4 Economic Aspects ............................................................................................................ 15
6.1.5 Financial Viability ............................................................................................................. 16
6.2 Understanding Evaluation Results by Comparison Matrices ............................................... 16
6.2.1 Comparison Matrices ...................................................................................................... 16
6.2.2 Decision Matrices ............................................................................................................ 17
7 M&V Process for Implementation of An EEDSM Program ............................................................ 17
8 References ..................................................................................................................................... 20
9 Appendix ....................................................................................................................................... 21
9.1 Appendix 1: Guidance on Sustainability Assessment .......................................................... 21
9.2 Appendix 2: Examples .......................................................................................................... 26
9.2.1 Engineering Aspects ........................................................................................................ 26
9.2.2 Environmental Aspects .................................................................................................... 27
9.2.3 Social Aspects .................................................................................................................. 28
9.2.4 Economic Aspects ............................................................................................................ 29
2

List of tables
Table 1 Program evaluation from engineering aspects ........................................................................ 12
Table 2 Program evaluation of environmental aspects ........................................................................ 14
Table 3 Program evaluation of social aspects ....................................................................................... 15
Table 4 Program evaluation of economic aspects ................................................................................ 16
Table 5 An example of a comparison matrix ......................................................................................... 17
Table 6 Calculation of total score for program evaluation ................................................................... 17
Table 7 Guideline on sustainability assessment ................................................................................... 26
Table 8 Example for program evaluation from engineering aspects .................................................... 26
Table 9 Example for program evaluation from environmental aspects ............................................... 28
Table 10 Example for program evaluation from social aspects ........................................................... 28
Table 11 Example for program evaluation from economic aspects ..................................................... 29
3

1 Introduction
The South African economy is experiencing a generally healthy growth period that has been reflected
in consistently high national electricity demand growth. As a result, Eskom is faced with a fast
diminishing reserve margin across the load profile. The surplus capacity under which Eskom
operated within the last several years has been largely depleted and Eskom’s power system will
remain tight while new capacity is built and old capacity is reinstated. In recent years, energy
efficiency has proven its worth in being able to limit the capacity needed through managing the
demand.
Current planned capacity expansion initiatives are not expected to meet the demand requirements
cost effectively within the required timeframes. Planning has therefore been reconsidered and a
comprehensive strategy to address demand, incorporating both supply and demand management
solutions, has been developed. Comprehensive energy efficiency (EE) and demand side management
(DSM) programs are led by ESKOM which will reduce Eskom’s peak demand supply requirement by
approximately 3,000 MW by March 2013 and a further 5,000 MW in the subsequent 13 years to
March 2026 in order to alleviate the imminent supply constraints and to displace the need for more
costly supply options currently under consideration. EEDSM interventions have been proven very
successful in assisting (on a much smaller scale) with the alleviation of the short term capacity
constraints during the 2006 Western Cape power crisis.
In order to further promote EEDSM activities, an important strategy is to evaluate existing EEDSM
projects and programs, recognize achievements, so that the impact of the EEDSM measures and
further improvement opportunities can be identified. Existing evaluation on EEDSM programs are
usually based on the energy and power saving, with an emission reduction assessment based on
given conversion ratios from energy consumption to greenhouse gas (GHG). This environmental
evaluation based only on GHG emission is obviously not enough since there are many environmental
aspects need to be addressed. Furthermore, recent research indicates that when the technological
EEDSM measures reach their energy saving limit, there is still the potential of further energy savings
when the social behaviour changes are promoted. For any EEDSM program, people are also
interested to find out the corresponding social and economic impact, for instance, how many new
jobs created each year, how is it aligned with the strategic positioning and restructuring of national
economy, etc. Therefore, the EEDSM program can provide more energy savings from social
behavioural approaches, and the impact of this program can be evaluated not only from
conventional engineering point of view, but can also be evaluated from comprehensive
environmental, social, and economic aspects.
Such a comprehensive evaluation for EEDSM program can be established with the help of a new
energy efficiency classification approach which classifies general energy efficiency in terms of the
efficiencies of performance, operation, equipment and technology (POET) [15‐17]. Note that
conventional evaluation on EEDSM program is based on the measurement and verification (M&V)
guidelines [6,7], and M&V teams with a strong engineering background are contracted by ESKOM for
the M&V of EEDSM projects. With the help of the POET framework, this document will present this
comprehensive evaluation guideline for EEDSM programs so that the existing M&V teams and
professionals with strong engineering background are still able to evaluate EEDSM programs not only
4

from the engineering and environmental point of view, but also the social and economic point of
view.
2 International EEDSM Evaluation—A Measurement and Verification
Approach
According to [12], physical evidence to evaluate an EEDSM program needs to be collected so that
calculations based on measured quantitative data can give a rigorous evaluation on this EEDSM
program, which is the so‐called M&V process that consists of mainly the baselining, performance
assessing and tracking procedures. Therefore, M&V for EEDSM program evaluation will be discussed
in this document.
[1], [6], [7], and [14] focus primarily on the evaluation of energy/power savings, with also, but
relatively less, emphasis on emission reductions. These documents provide comprehensive M&V
methodologies for the evaluation of engineering aspects of EEDSM programs, particularly [14]
provides detailed M&V plans for different scenarios such as lighting systems, motors, chillers,
geothermal heat pump, water, and renewable projects.
Besides the engineering aspects of energy and power consumptions, a full evaluation on
environmental, social, and economic aspects need also be evaluated for the EEDSM programs. [12]
lists many protocols which includes not only the general energy/power saving impact, but also the
market effects evaluation where the change of market structure or behaviour of market participants
with respect to an increase in the adoption of energy efficient measures in the EEDSM program is
evaluated. [3] and [4] focus more on the economic analysis of DSM projects.
[8] provides the monitoring and evaluation technique for forestry projects in carbon emission
reduction. Environmental and socioeconomic impacts of an EEDSM project are also briefly
discussed in [8], where the impacts on the following factors are mentioned: dams and reservoirs,
hazardous and toxic materials, indoor air quality, industrial hazards, insurance claims, occupational
health and safety, water quality, wildlife and habitat protection or enhancement, cultural properties
(archaeological sites, historic monuments, and historic settlements), distribution of income and
wealth, employment rights, gender equity, induced development and other sociocultural aspects
(secondary growth of settlements and infrastructure), long‐term income opportunities for local
populations plants (jobs), public participation and capacity building, quality of life (local and
regional).
The European Union (EU) project in [2] focuses on the socio‐economic impacts on renewable
energies, for instance the impact of renewable energy on social welfare, migration flows, technology
status, culture, security of energy supply, and environment are discussed. [13] discusses the socio‐
economic measurement and validation for pilot building energy efficiency projects in several EU
countries, where psychological model of human behaviour, human‐machine interaction, and some
socio‐economic evaluation indicators are discussed. [10] summarises lessons learnt from Denmark
practice on socio‐economic assessment of wind power systems. There are also discussions written in
Danish language on the effects of Danish energy policy on employment and export [9] which are not
included here.
These international guidelines and practices provide very helpful discussions on the general EEDSM
evaluations, however, none of them provides a full discussion on the evaluation of all the
engineering, environmental, social, and economic evaluation of EEDSM programs at an operational
level. To resolve this problem, this document will provide an operational guideline on the full
5

evaluation of EEDSM programs from all the engineering, environmental, social and economic
aspects.
3 The South African Context
South Africa, like other countries in the world, struggles to provide enough energy to sustain the
growing economy, leading to increasing costs, and increased damage to the natural environment. It
is necessary to face not only the direct consequences of any project, but also the broader impacts on
the economic, social and natural environment. The evaluation of EEDSM from all these aspects are
realized by the South African government, and in many energy related projects, such evaluation
criteria as emission reduction, job creation, human capacity building, etc, have become important
factors in the determination of funding support and achievement recognition. [7] and [11] are also
developed to evaluate the engineering and emission reduction indicators of EEDSM projects. The
development of a comprehensive evaluation scheme and the corresponding implementation need
the participation and help of all stakeholders.
4 EEDSM Program Evaluation Guideline
This program evaluation guideline will provide guidance on the evaluation of EEDSM programs from
the viewpoint of a broad classification of energy efficiency, and thus provide the stakeholders of the
programs a broad scope of evaluation from the engineering, environmental, social, and economic
aspects. The key tool applied in this guideline is the classification of energy efficiency in terms of
performance, operation, equipment, and technology (POET). The POET model will be explained in the
next section. With the help of this POET classification, all the expected factors including energy and
power saving, energy cost saving, carbon emission, job creation, and economic growth can be
evaluated.
This guideline will aid program managers, program evaluation panel, and the M&V practitioners to
understand the wealth of factors affecting the EEDSM program and to prioritise important energy
efficiency factors in the design or evaluation of the program. For example, this guideline will help
program managers to design EEDSM programs and prepare program reports in terms of the
evaluation criteria; provide a flexible evaluation framework for program evaluators; and cast a new
light on the evaluation of social, economic, and environmental issues for M&V professionals.
4.1 Contents of Program Evaluation
The content of a program evaluation can cover many aspects, for instance, market needs
assessments, process evaluations, retrospective outcome/impact assessments, and cost‐benefit
evaluations. These types of evaluation studies help managers determine if timely adjustments are
needed in program design or implementation to improve the rate, or quality of achievements
relative to the committed resources. Program evaluations are in‐depth studies of program
performance and customer needs. They can be used to produce information about the linkage
between program performance and resources and about how to improve performance. The benefits
of conducting an evaluation are numerous. For example,
1) They can help to estimate how well the program is achieving its intended objectives
2) They help to improve the program development and implementation
6

3) They quantify results and cost‐effectiveness, as necessary, to help better communicate the value
of the program.
Some key definitions are described below:
Program: A project or group of projects with similar characteristics and installed in similar
applications, where multi‐faceted effects are observed and evaluated.
Evaluation: The performance of studies and activities aimed at determining the effects of a program;
any of a wide range of assessment activities associated with understanding or documenting program
performance, assessing program or program‐related markets and market operations; any of a wide
range of evaluative efforts including assessing program‐induced changes in energy efficiency
markets, levels of demand or energy savings, environmental impacts, social and economic impacts
and program cost‐effectiveness.
Program evaluations: Program evaluations are systematic and objective studies, conducted
periodically or on an ad hoc basis, to assess how well a program is achieving its intended goals. These
evaluations have a retrospective focus, with a view to assessing past performance and developing
recommendations for improvements, with an exception of the evaluations of market needs that can
have a current or prospective focus. Some evaluations usually require certain level of details in data
collection and analytical methodology that goes beyond routine performance‐monitoring reporting.
This helps the decision makers determine what kinds of timely adjustments may be needed in
program design or implementation to improve the rate or quality of achievement relative to the
committed resources. It is not necessary to have in‐depth familiarity with these methods to benefit
from a general program evaluation, but program managers need to have enough familiarity to select
and monitor an evaluation contractor who will make decisions about evaluation methodologies.
The POET guideline:
1) Explains how to establish an evaluation marking system with the help of the POET classification
approach
2) Provides guidance on how to evaluate a program from not only engineering aspects, but also
social, economic, and environmental aspects; and
3) Gives a vision of future M&V projects to evaluate social, economic, and environmental issues.
4.2 Purpose
This program evaluation guideline should serve two critical purposes – program improvement and
accountability. Many evaluations will be designed to serve both of these purposes.
The purpose of this guide is to create and manage objective, high quality, independent and useful
program evaluations. The guide could be used by those without prior training or experience in
program evaluation and should make it easier for them to take advantage of this useful and
increasingly required program‐management tool. Such a guide is intended for use by all stakeholders
involved in any EEDSM program.
This guide does not answer all of the technical questions about evaluation and M&V methodology,
but it will provide enough information to help program managers to:
Identify the questions that they need to answer to fulfil a general program evaluation
Monitor the evaluation progress
Implement credible quality assurance (QA) controls
7

Ensure that the evaluation report presents useful findings and recommendations
Ensure that the findings are disseminated to those who need them.
4.3 Benefits
The two main benefits of conducting evaluations of energy programs are: 1) to reliably document
program effects and 2) to improve program designs and operations to be more cost‐effective in
obtaining energy resources.
These goals serve to provide an evaluation framework that when implemented in South Africa will:
Provide reliable evaluation results to support energy policy and supply decisions,
Allow programs to be equably compared according to their energy impacts,
Help understand and verify program energy and peak savings,
Help identify and quantify market and non‐energy effects,
Provide information needed to estimate program cost‐effectiveness, and
Provide recommendations for program changes that help improve cost‐effectiveness.
In addition to accomplishing the above high‐level goals, the evaluation results help to accomplish or
support the following more specific objectives:
Increase the level of reliability of impact estimates of program savings for use in resource
planning forums where the uncertainty of these estimates needs to be compared against the
uncertainty of other key components of the resource plan.
Increase the quality of feedback to program administrators from evaluation projects to both
improve program designs and increase the net savings from their programs.
Provide to stakeholders not only energy savings and carbon emission reduction, but also
socio‐economic impacts from the program such as job creation, poverty alleviation,
economic growth, etc.
Provide guidance to DSM and ESCos on the methodological approaches and study focus
needed to perform specific types of evaluations.
Provide a framework with flexibility that allows for the use of alternative evaluation
approaches, especially when they can be shown to provide results as reliable as the methods
presented in the guideline.
5 Scope of Work (SOW)
Program evaluation requires human technical expertise and experience. Specific projects may also
require specific expertise; however, in this scope of work it is proposed that a universal method
could be used to ensure that holistic evaluation projects are undertaken, considering not only the
engineering aspect, but also the economic, social and environmental aspects that would increase the
overall sustainability of the proposed project.
Program managers, evaluation panel, and M&V practitioners need to have some background in the
goals, objectives and constraints of a proposed EEDSM program or project. At the program level, it
would be possible to synthesise the projects to such an extent that the holistic approach of each
project could be assessed and directed.
8

5.1 Objective
The purpose of this guideline is to provide to both experienced and inexperienced program
managers, program evaluators, and M&V practitioners with tools to evaluate holistic evaluation
programs. Holistic evaluation programs not only include the engineering changes to make a project
more energy efficient, but also focus on the economic, social and environmental issues that would
support a sustainable project. These measurements will help to synthesise information in order to
effectively review approaches, standardise reviews and enable centralised tracking of evaluation
parameters.
For a program to be sustainable, it has to meet certain sustainability criteria:
1) Be supported by a good organisational structure, which includes, for instance, relevant
policies, regulations, incentives, competitions, awards, penalties, and human sensitisation;
2) Have a win‐win compliance among all the engineering, financial, social, and economic
performance indicators; and
3) Have necessary engineering support.
Four main aspects of an efficient system will be considered in a program evaluation: engineering,
economic, social and environmental aspects.
The difficulty is always to decide the balance between the various aspects in order to come up with a
feasible project, taking these four important aspects into consideration. The objective of this guide is
to indicate to program managers and evaluators alike, the basic and most prominent issues that need
to be addressed.
5.1.1 POET Efficiency
For the purpose of this guideline, measurement of energy efficiency is summarized to have the
following four components: performance efficiency (P), operation efficiency (O), equipment
efficiency (E), and technology efficiency (T). This POET classification maintains energy efficiency at its
broadest possible scope, taking all aspects of efficiency into consideration. The four components of
efficiency will be discussed below. Technology efficiency will firstly be discussed, since all other
efficiency components depend on the types of technology used.
It is important to note again that the sustainability of an energy program should include not only the
engineering aspects of energy efficiency, but will also include economic, social and environmental
feasibility.
5.1.1.1 Technology efficiency (T)
Technology efficiency is a measure of efficiency of energy conversion, processing, transmission, and
usage; and it is often limited by natural laws such as the energy conservation law. Technology
efficiency is often evaluated by the following indicators: feasibility; life‐cycle cost and return on
investment; and rates of energy conversing/processing/transmitting.
Technology efficiency is characterized by its novelty and optimality. On the one hand, ground
breaking and feasible novel technologies often defeat older peers and occupy the market quickly. On
the other hand, these novel technologies always challenge optimality through the pursuit of scientific
limits and the quest for new possible extremes.
5.1.1.2 Equipment efficiency (E)
Equipment efficiency is a measure of the energy output of isolated individual energy equipment with
respect to given technology design specifications. The equipment is usually considered being
9

separated from the system and having little interactive effect to other equipment or system
components. Equipment efficiency is evaluated by considering the following indicators: rated
capacity; specifications and standards; constraints; and maintenance.
Equipment efficiency is specifically characterized by its standardization and constant maintenance.
The most important aim of equipment efficiency is to minimize the deviations of the actual
equipment parameters to the given design specifications. The difference between equipment
efficiency and technology efficiency is easily illustrated by considering the compact fluorescent lights
(CFL) example: The study on the improvement of CFL technology to provide more efficient lighting
facilities forms part of the category of technology efficiency improvement, while replacing
incandescent lights with CFLs is part of the category of equipment efficiency improvement.
5.1.1.3 Operation efficiency (O)
Operation efficiency is a system wide measure which is evaluated by considering the proper
coordination of different system components. This coordination of system components consists of
the physical, time, and human coordination parts. These parts can again be indicated by sizing,
matching, skill levels, and time control of these system components. Operation efficiency has the
following indicators: physical coordination indicators (sizing and matching); time coordination
indicator (time control); and human coordination. In particular, sizing of a single system component
is to consider the relationship of this component with respect to the rest components of the system,
thus sizing of the system component is an operational issue comparing with the capacity
consideration in the equipment efficiency context. Automatic driving is an example of ‘matching’.
When a car is driven through different speed restriction zones and different traffic flows, the
automatic driving system must determine the different speeds of the car for different road
conditions to minimize its fuel consumption, or equivalently to maximize its operation efficiency.
5.1.1.4 Performance efficiency (P)
Performance efficiency of an energy system is a measure of energy efficiency determined by
external but deterministic system indicators such as production, cost, energy sources, environmental
impact and technical indicators amongst others. The following lists some general indicators for the
evaluation of performance efficiency.
• Production: Production is often determined by the market, and the performance efficiency can
change whenever market conditions change.
• Cost: The change of the cost of a process will give rise to the change of its performance efficiency.
For example, when a time‐of‐use (TOU) electricity tariff is introduced, an end user often tries to shift
the load from peak time to off‐peak time period, resulting in the corresponding electricity cost being
reduced, and thus the performance efficiency improved.
• Sources: When an energy system consists of different forms of energy sources, for example,
electricity, gas, coal, fuel and wood, amongst others, the performance efficiency of the system can be
evaluated by considering the usage of these different sources. For instance, using gas in some
circumstances will have better performance efficiency than using coal.
• Technical indicators: Some technical indicators are used as a means to measure aspects of
performance. At other times, technical indicators may be built into the performance objective to
drive the design process.
It is worth noting that sometimes these performance efficiency indicators are contradictory or in
competition with each other. Any system will usually be expected to maximize the production and at
the same time minimize cost, emission and social impact. Therefore the performance efficiency can
only be improved when certain trade‐offs among different indicators are made. The sustainability of
10

the energy system could be reached when the engineering indicators (e.g. sources) do not compete
with the social, economic or environmental indicators (e.g. production, cost, environmental
concerns).
6 EEDSM Program Evaluation Criteria and Methodology
The program evaluation criteria and methodology are developed in such a way that they will assist
program planners, managers and evaluators to organise, design, and implement retrospective impact
evaluations that:
Set holistic and realistic goals for the program.
Pragmatically assess progress toward the key goals of the program.
Focus on the most important issues identified before the commencement of the program or
its evaluation.
Focus on the aspects that program planners and managers can control and/or influence.
Give credit that is due to program managers for the direct and indirect effects clearly
attributable to their programs.
Produce credible evaluations.
For this purpose, the following general steps for program evaluation are advised, and the POET based
process evaluation criteria and methodology are developed. It is important to go through the
following 6‐setep process with a view of what the POET model entails. This will equip the program
manager a so‐called ‘bird’s‐ eye view’ of the project or program.
Step 1 – Review program descriptions
Step 2 –Design the evaluation tables to highlight and prioritize important issues to be evaluated in
the program, and issue the program evaluation criteria to all stakeholders
Step 3 – Program managers (ESCos) prepare program documentation according to evaluation criteria
Step 4 – Program evaluation panel evaluates program reports and supporting documents and data
Step 5 – Finalize evaluation report and issue report to all stakeholders
Step 6 – Ensure actions taken on key issues through performance tracking
A general guideline often addresses issues such as:
Regulatory requirements
Energy and demand impacts
Economic/cost/financial impact
Social impact e.g. health benefits; job creation; etc.
Environmental impacts
Benefits to all stakeholders e.g. NERSA, Eskom, customers
General matters to all existing technologies which should evolve with time
Considerations of load shedding, double counting, kick back, etc.
Each of these falls into either technical, environmental, social or economic aspects those need to be
considered. Select and define specific indicators and their parameters (see Appendix 1 for examples
from the CDMGS toolbox). Goals for each of these should be broken down into sections, using the
POET model in order to ensure that the different aspects of the goal and how it interacts with other
goals will be taken into consideration.
11

6.1 Evaluation Marking System Development
Following the assessment methods suggested above, all the necessary aspects will be considered and
prioritised in such a way that a sustainable solution could be suggested. The use of a decision matrix
that incorporates the above four major aspects of any energy efficiency project and an analysis of
this matrix according to the various indicators of the POET structure are highly suggested. The matrix
should include several indicators from each of the four important aspects to be considered.
6.1.1 Engineering Aspects
Table 1 shows such a matrix for the evaluation of engineering aspects of a program. Technical issues
within a program will be evaluated by one or more of the T, E, O, P indicators in terms of the impact
from the program. For instance, the energy consumption before and after the program and the
expected/actual saving, the percentage of renewable energy added in the energy system or grid,
etc., can be evaluated by purposely selected POET indicators. A score will be given for each of the
evaluated items, using a fixed indicator. Weighting factors will be determined for each of the
evaluation indicators to indicate the preference or importance of such indicators in the evaluation.
The subtotal score for engineering aspects will be the sum of all the products of the scores and the
corresponding weighting factors. An example is provided in Appendix 2 to further illustrate this
evaluation process. The program evaluator can select a number of POET indicators for evaluation,
and then either give the corresponding scores for items evaluated, or design a number of Yes/No
questions to ask and determine the corresponding scores according to the Yes/No answers. Note
that existing M&V guidelines provide enough technical details in the evaluation of the energy or
power indicators in different scenarios, therefore, the details of the program evaluation from
engineering aspects can follow the general guidelines and techniques in references [3], [4], and [6],
and the evaluators can design flexibly the evaluation criteria according to these general guidelines
and the specific program needs.
Engineering aspects Technology Equipment Operation Performance
Technical issues of a
program (e.g. processes
of energy auditing,
planning, and targeting;
methodologies; etc.)
Items which might be
considered in scoring (e.g.
pre‐ and post‐
implementation data and
impact from the program
(savings, etc.))
Scores (out of 100)
Weighting factors (in
percentage)
Subtotal score
Feasibility;
Life cycle
cost;
Energy
transfer
/transmitting
/converting
ratio
Specifications;
Maintenance;
etc.
Table 1 Program evaluation from engineering aspects
12
Sizing;
Matching of
system
components;
Energy/power
consumption;
Energy
carrier/source;
Energy safety;
Reporting;
Project
organisation

6.1.2 Environmental Aspects
Environmental aspects of the program can be evaluated similarly as the engineering aspects, which
are set out in Table 2. Issues can be evaluated including air, water, atmosphere,
biodiversity/ecosystem degradation, and land use amongst others. Each issue will be evaluated by
certain criteria from properly chosen POET related indicators. For example, the criteria to evaluate
air can be the emissions of NOx and SOx, particulates, NMVOCs, heavy metals including mercury,
radio nuclides, etc. Each criterion can be further broken down in terms of P, O, E, T indicators in case
that the program stakeholders need these detailed evaluations. For instance, the emission of NOx
and SOx can be considered from the POET perspective such as the amount of emissions (P); the
interactions of the NOx and SOx emission with other system components including CO2, production,
energy consumption, etc. (O); the absorption process and equipment for NOx and SOx (E); and the
mechanism which cause the NOx and SOx emission and the relevant technology adopted to reduce
or absorb the NOx and SOx emission (T). Similarly, for the evaluation criteria of other issues such as
water and atmosphere the POET indicators can be applied again. Program evaluators need to choose
evaluation criteria which meet best the purpose and content of a program. The following are
examples of evaluation criteria.
Evaluation criteria for water cover direct water consumption of plant, water use in fuel chain,
contamination at plant and in fuel chain including acid mine drainage (AMD) to aquifers;
Evaluation criteria for atmosphere can cover greenhouse gas (GHS) emissions in the entire
production cycle (including fuel chain, transport of fuel or sorbent as well as uranium
processing and enrichment done in South Africa);
Evaluation criteria for biodiversity or ecosystem degradation will be land degradation (land
pollution via air or water is not double counted here, e.g. acid rain involves additional impact
of SOx and NOx, beyond air pollution impacts on human health); and degradation of water
catchments (including wetlands; impacts on sensitive ecosystems and species);
Evaluation criteria for land use will consider footprint of plant (e.g. in the project area of in
wind farms , turbines and access roads have a footprint about 5% ‐10% of the total project
area and solar farms could accommodate agricultural activity); footprint of fuel supply
including land for mining (while eventual rehabilitation is a theoretical possibility, other
productive utilisation is precluded for a substantial period).
Note that an environmental impact assessment needs to consider cumulative impacts of the factors
evaluated in the energy system, while those factors which do not have cumulative impacts can be
ignored. Visual, noise and dust pollution are not included as they do not accumulate in the same way
as impacts considered above, they are much more site‐specific and visual impact is highly subjective;
it is considered that such impacts will be adequately covered by Environmental Impact Assessment
and Environmental Management Plans, and do not serve for significant differentiation amongst
technology or scenario options.
Note further that some programs focus only on electrical energy savings, and usually the energy
consumption will be a performance indicator to evaluate these programs. These programs also have
positive impacts to the environment and the reduced CO2, SOx, and NOx can be calculated from the
amount of energy saved, thus these emission indicators should not be double counted in the
corresponding program evaluation. There are also programs with the only purpose of emission
reduction. For these programs, the performance indicators such as the amount of CO2, SOx and NOx
emitted into the air will play a key role in the evaluation and thus are highly weighted in calculating
the scores, while the energy consumption indicator will play a less important role and will be
weighted much less. For programs which focus on both the energy saving and emission reduction,
13

the energy saving and emission related performance indicators will be evaluated, and the weighting
factors are chosen according to particular program descriptions.
Environmental aspects Technology Equipment Operation Performance
Environmental issues (e.g.
air, water, atmosphere,
biodiversity/ecosystem
degradation, land use, etc.)
Items which might be
considered in scoring (e.g.
pre‐ and post‐
implementation data and
impact from the program
(savings, etc.))
Scores (out of 100)
Weighting factors (in
percentage)
Subtotal score
6.1.3 Social Aspects
Table 2 Program evaluation of environmental aspects
Again, social aspects could also be assessed using the POET model. The matrix related to social
aspects is shown in Table 3, which can accommodate aspects such as the number of jobs created, the
quality of the created jobs, the health safety and security aspects in the workplace, knowledge levels,
access to health care, access to water and sanitation, quality of life of the employees and community
in and around the applicable plant.
The criteria could be assessed according to the POET model, for instance, instead of counting the
absolute number of jobs created as in the past, the quality of the created job could be assessed. The
workplace environment and available resources should be assessed (P); the compatibility of this job
with national skills development and training initiatives (O); the individual position’s job description
should be clear to both the employer and employee (E); and the creation of new types of jobs should
be considered (T). Evaluation criteria of other issues could include:
Governance. Transparency of contracting and liability, especially when issues are
underwritten by fiscus; the extent of requirements for regulation (primarily of plant
operation, but including enforcement of EMPs for mines and provisions for
decommissioning); requirements for restrictions on access to information; lack of
transparency of industries / technology vendors;
Stakeholder and community participation: compatibility with local community participation
(including as land reform beneficiaries); compatibility with other local productive activities;
potential for sweat‐equity; decentralisation and devolution from monopolies.
Intergenerational equity: depletion of natural resources / natural capital (domestic);
exposure to long term supply constraints on finite resources (recognising international peak
supply prospects); lock‐in to technologies with heavy ecological footprint;
Health, Safety and Security: burden on law enforcement and the extent of security
requirements (including transmission, specifically if electricity is imported); the burden on
public safety services and the provision for emergency response; vulnerability to
and potential for sabotage.
14

It is important to consider the social impacts and other social contributors for each project planned.
The identified issues might not all have an equal priority, but weighting would assist in delineating
the most important issues to be taken up into an actual social survey.
Social aspects Technology Equipment Operation Performance
Social issues of a program
(e.g. employment,
governance, stakeholder
and community
participation, health,
safety and security,
Intergenerational equity,
etc.)
Items which might be
considered in scoring (e.g.
pre‐ and post‐
implementation data and
impact from the program)
Scores (out of 100)
Weighting factors (in
percentage)
Subtotal score
6.1.4 Economic Aspects
Table 3 Program evaluation of social aspects
Economic aspects should be assessed in a similar fashion suggested in the above. Saving energy and
simultaneously improving production would yield economic improvements. The economic aspects
that could include balance payments of investments, life‐cycle generation costs, local content of the
system and assessments of the risk and prospects of a specific project.
Each issue will be evaluated by certain criteria from properly chosen POET related indicators. For
example, the local content could be assessed as such: proportion of local content now, or for the first
plant (T); prospects for increasing local content of a future plant (this is particularly pertinent to the
scenario as a whole are technologies to be deployed at sufficient rate to justify local manufacture)
(E); dependence on imported parts and systems (O); and the type of technology use in the plant (P).
Other such examples include:
Costs: life‐cycle generation costs; other system infrastructure
Financial / Price risk: probability of cost over‐runs subsequent to contracting (track record of
technology); exposure to fuel price volatility and escalation over medium to long term (short
term covered under security of supply); exposure to carbon pricing/border taxes (note: this is
not double‐counting GHGs as it does not relate to the impact of atmospheric pollution);
state/public exposure to liability;
Financing prospects: eligibility for development and/or climate finance;
prospects/attractiveness for private equity; compatibility with long‐term national savings
(e.g. investment by pension funds);
15

Strategic positioning and restructuring of national economy: Potential for global leadership
and export of key energy technology(ies); contribution to emerging Green Economy Strategy;
alignment with broader transition to a low carbon economy;
Economic aspects Technology Equipment Operation Performance
Economic issues of a
program (e.g. Cost, local
content, financial/price
risk, financing prospects,
strategic positioning and
restructuring of national
economy, etc.)
Items which might be
considered in scoring (e.g.
pre‐ and post‐
implementation data and
impact from the program
(savings, etc.))
Scores (out of 100)
Weighting factors (in
percentage)
Subtotal score
6.1.5 Financial Viability
Table 4 Program evaluation of economic aspects
Assessing the financial viability of a project or a program is primarily concerned with the direct
project costs and involves setting up a project plan and budget and determining the intended
investment in assets. Project income statements and balance sheets will aid the project manager to
assess the viability of the project in the usual way. Assessment of the financial viability of a project is
normally done by an expert well versed in all the contributing factors of an evaluation and we
recommend that experts be used in order to ensure that the assessment is enforceable.
6.2 Understanding Evaluation Results by Comparison Matrices
6.2.1 Comparison Matrices
Program evaluation panel can select important issues for evaluation, and prepare the above four
tables for evaluation, in conjunction with the financial viability study. After the scoring of the five
figures on engineering, environmental, social, economic, and financial aspects, a comparison matrix
can be further established to understand the output from the program. Table 5 serves as an example
of such a comparison matrix, where it can be read, for the impact from this program, that for every
MWh saved, there will be 20/500=0.04 job∙year created; and similarly, for every ton of CO2 reduced,
there will be 0.02 job∙year created. It shows also that the cost for each created job∙year is
R10000/20=R500. The figures in the table could be ones taken from the previous four tables and the
financial viability study. The figures in Table 5 are indicative.
16

Energy
saved (500
MWh)
CO2
reduction
(1000 tons)
Number of
jobs created
(20)
Economic
growth
(R25000)
Project cost
(R10000)
6.2.2 Decision Matrices
Energy
saved (500
MWh)
CO2
reduction
(1000 tons)
17
Number of
job∙year
created (20
job∙year)
Economic
growth
(R25000)
1/1 500/1000 500/20 500/
25000
1000/500 1/1 1000/20 1000/
25000
Project
cost
(R10000)
500/
10000
1000/
10000
20/500 20/1000 1/1 20/25000 20/
10000
25000/500 25000/1000 25000/20 1/1 25000/
10000
10000/500 10000/1000 10000/20 10000/
25000
Table 5 An example of a comparison matrix
After the subtotal scores of the program have been given from the engineering, environmental,
social, economic, and financial aspects, then each of the five subtotal scores are multiplied with a
corresponding weighting factor, and the products are summed together to find the total score for
the program. This process is illustrated by the following decision matrix in Table 6 which is used to
help the decision‐making of stakeholders.
Engineering Environment Social Economic Financial
Subtotal scores
Weighting
factors (in
percentage)
Total score
Table 6 Calculation of total score for program evaluation
The comparison and decision matrices are useful tools to compare and rank competing and
progressing projects. Care should be exercised when yes/no questions are used to qualify/disqualify
the project proposals. In such cases, a special notation could be used in the comparison and decision
matrices to denote the status, such as .
7 M&V Process for Implementation of An EEDSM Program
The above POET based EEDSM program evaluation methodology provides a general evaluation
guidelines for both the program evaluators and the program developers. Note that usually EEDSM
programs are developed by contracted energy analysts, or the so‐called Energy Service Companies
(ESCos). As the customer contracted program developer, an ESCo needs to make thorough
1/1

investigation and comes up with a feasible EEDSM program. The ESCo often claims certain impact
from the implementation of this EEDSM program. According to the evaluation criteria in the previous
section, the ESCo can develop the program and thus claim the relevant engineering, environmental,
social, and economic impacts of the program. Both the customer and ESCo want to know if these
claimed impacts have been achieved after implementation, therefore, as an independent third party,
the M&V team will help further to measure and verify these claimed impacts.
The M&V process for the EEDSM program is similar to usual energy saving M&V process in [6] as
explained below.
1) This program evaluation guideline document is distributed to ESCos or any other program
developer. The M&V team will award a final mark to each EEDSM program at the end of the
evaluation.
2) The ESCo will develop an EEDSM program from all the engineering, environmental, social,
and economic aspects; then the ESCo claims the corresponding impact of the program on
each evaluating factor, where the impact of each evaluating factor is claimed in either a
quantitative way or a qualitative way. Examples for quantitative achievements include the
exact amount of energy saving to be achieved, the number of jobs created each year, the
reduced amount greenhouse gas emission, etc. Examples for qualitative achievements can be
statements on how the program is aligned with national economic strategic positioning, why
the program is compatible with local participation, etc.
3) The ESCo submits the M&V request to the M&V governing body (e.g. ESKOM Energy Audit),
then the governing body will allocate this request to an M&V team.
4) The M&V team prepares the scoping report to describe the overall program after the
necessary communication with the ESCo and customer.
5) The M&V team prepares the M&V plan report which needs the sign off from both the ESCo
and the client. This M&V plan includes not only key parameters to be monitored, metering
plan, but also the evaluating and marking criteria. The agreement among the ESCo, the
customer, and the M&V team must be reached on the evaluating and marking criteria. For
example, the three parties need to determine which factors need to be evaluated and the
corresponding weight that each factor occupies in the total mark; the percentages of
crediting/debiting marks awarded for over/under performing; etc.
6) The M&V team issues the baseline report according to the M&V plan, this baseline report
needs also the agreement from both the ESCo and the client. The baseline report is based on
the data and information collected from the project sites before the implementation of the
EEDSM program. Meters need to be installed to quantify baseline information in engineering
indicators, and surveys and site visit will be performed to confirm other quantitative and
qualitative baselines (or baseline marks) for other evaluation indicators. A baseline
information or baseline mark will be given for the existing situation within the project
boundary.
7) The EEDSM is implemented, and the M&V team issues the post implementation certificate to
confirm the implementation.
8) The M&V team issues the performance assessment report to evaluate if the claimed impact
has been achieved. Usually this performance assessment will be issued at least once and
usually three times. A mark will be awarded to the assessed project in each assessment
report. The success of a program will depend on if the post‐implementation mark is higher
than the baseline mark. Baseline mark adjustment might be needed if necessary. The
principle for baseline mark adjustment is the same as the energy saving baseline adjustment
principle in [6].
9) The M&V team will issue a series of performance tracking reports to continuously monitor
the impact from the implementation of the program after the performance assessment
18

eporting period. Details on the frequency and the number of performance tracking reports
can be determined by the needs of the customer and ESCo.
Important to note that although the evaluation on the social and economic aspects of an EEDSM
program brings a lot of new exciting evaluating factors in the M&V projects, existing M&V
professionals will still be able to handle the new challenge since the evaluation process is exactly the
same as the current energy saving M&V projects. For the practical operation of the M&V for EEDSM
programs, necessary training on the study of this evaluation guideline for both M&V professionals
and ESCos is strongly advised.
19

8 References
[1] American Society of Heating, Refrigeration and Air Conditioning Engineers, Measurement of
Energy and Demand Savings, Guideline 14‐2002, Atlanta, Georgia, 2002.
[2] M.A. Aguado‐Monsonet, Evaluation of the socio‐economic impacts of renewable energies:
Global survey to decision‐makers, European Commission, Joint Research Centre, Institute for
Prospective Technological Studies, Spain, 1998.
[3] California Public Utilities Commission, California Standard Practice Manual: Economic analysis of
Demand‐Side Programs and Projects, 2001.
[4] W. C. Clark II, A. Sowell, and D. Schultz, The standard practice manual: the economic analysis of
demand‐side programs and projects in California, Int. J. Revenue Management, 2002.
[5] Ecofys, TÜV‐SÜD and FIELD, Guidance on Sustainability Assessment (Gold Standard Toolkit 2.0),
July 2008, p. 126‐130.
[6] Efficiency Valuation Organization, International Performance Measurement and Verification
Protocol, vol. 1, 2010.
[7] Eskom, The Measurement and Verification Guideline for Energy Efficiency and Demand‐Side
Management Projects and Programmes, 2009.
[8] Lawrence Berkeley Laboratory, Best Practices Guide: Monitoring, Evaluation, Reporting,
Verification, and Certification of Climate Change Mitigation Projects, United States Agency for
International Development, November 2000.
[9] H. Mæng, Beskæftigelses‐ og Eksportvirkninger ved Dansk Energipolitik – Eksemplificeret ved
vindkraft, solvarme, biogas samt decentral kraftvarme på naturgas (Effects of Danish Energy
Policy on Employment and Export – Exemplified in the case of wind power, solar heating,
biogas and decentralised combined heat and power using natural gas), Aalborg University,
Department of Development and Planning, Instituttets skriftserie nr. 223, 1998.
[10] J. Munksgaard and A. Larsen, Socio‐economic assessment of wind power‐lessons from Denmark,
Energy Policy, vol. 26, no. 2, pp. 85‐93, 1998.
[11] South African Bureau of Standards, Measurement and Verification of Energy Saving, Technical
Specification, SABS Standards Division, ISBN: 978‐0‐626‐23668‐7, 2010.
[12] TecMarket Works Team, California Energy Efficiency Evaluation Protocols: Technical,
Methodological, and Reporting Requirements for Evaluation Professionals, California Public
Utilities Commission, April 2006.
[13] University of Saarland USAAR, Specification of the methodology for the measurement and
validation of the socio‐economic impact of the pilots, D1.3, Best Energy Project, December 2009.
[14] USDOE, M&V Guidelines: Measurement and Verification for Federal Energy Projects, U.S.
Department of Energy, Office of Energy Efficiency and Renewable Energy, Version 2.2, 2000.
[15] X Xia and J Zhang, Energy Efficiency and Control Systems‐from a POET perspective, IFAC
Conference on Control Methodologies and Technology for Energy Efficiency, Portugal, 29‐31
March 2010.
[16] X Xia and J Zhang, Energy Auditing—from a POET Perspective, International Conference on
Applied Energy, Singapore, 21‐23 April 2010.
[17] X Xia and J Zhang, Modeling and Control of Heavy Haul Trains, IEEE Control Systems Magazine, to
appear in August 2011, doi: 10.1109/MCS.2011.941403.
20

9 Appendix
9.1 Appendix 1: Guidance on Sustainability Assessment
The following descriptions on key issues in Program Evaluation taken from [5] are helpful for
program evaluators to determine which issues will be needed for the evaluation of a particular
program.
Issues Description Possible parameters
Environment
Air quality Air quality refers to changes compared to the
baseline in:
Pollution of indoor and outdoor air which may
have a negative impact on human health or
the environment, including particulates, NOx,
SOx, lead, carbon monoxide, ozone, POPs,
mercury, CFCs, Halons. Also odour is
considered to be a form of air pollution.
Water quality
and quantity
Pollution with gases covered under the Kyoto
Protocol (carbon dioxide (CO2), methane
(CH4),
Nitrous oxide (N2O), hydrofluorocarbons
(HFCs), perfluorinated carbons (PFCs) and
sulphur hexafluoride (SF6).) Are not included in
this category as this category refers to changes
in the environment in addition to reductions of
greenhouse gases since GHG reductions are
included in all greenhouse gas reduction
projects by definition.
Water quality and quantity refer to t changes
compared to the baseline in:
Release of pollutants and changes in water
balance and availability in ground‐ and surface
water and its impacts on the environment and
human health, including biological oxygen
demand and chemical oxygen demand,
thermal pollution, mercury, SOx, NOx, POPs,
lead, coliforms (bacteria from animal waste)
Soil condition Soil condition refers to changes compared to
the
baseline in:
Pollution of soils, pollution of soils can be
21
NOx
SOx
Lead
CO
Ozone
POPs
Mercury
CFCs
Halons
Respirable Suspended
Particulate Matter
(RSPM)
NH3
SO2
NO2
PM10
VOC
Total Suspended
Particulate Matter
(TSPM)
Levels of :
Biological oxygen demand
Biochemical oxygen
demand
Thermal pollution
mercury
SOx
NOx
POPs
lead
coliforms (bacteria from
animal
waste)
Levels of :
Lead
SOx
NOx

Other
pollutants
caused by lead, SOx, NOx, mercury, cadmium,
possibly combined by a negative
corresponding impact on human health.
Organic matter content
Erosion level
This indicator refers to changes compared to
the
baseline in:
Other pollutants of the environment which are
not already mentioned. For instance level of
noise/ light, frequency of noise/light and time
occurrence (daytime/night‐time, weekdays/
weekend) are relevant for consideration.
Biodiversity Contribution to biodiversity refers to changes
compared to the baseline in:
Number of genes (i.e., genetic diversity within
a species) species and habitats existing within
the project’s impact boundaries.
Alteration or destruction of natural habitat
Depletion level of renewable stocks like water,
forests, fisheries
22
mercury
cadmium
Level of noise
Frequency of noise (per
day, per week, per
month)
Time
occurrence(day/night,
weekdays/weekend)
Number of affected
and/or
threatened plants
Number of affected and
/or
threatened mammals,
birds,
reptiles, fishes, and other
species and habitats
Issues Description Possible parameters
Social development
Quality of
employment
Livelihood of
the poor
Quality of employment refers to changes
compared to the baseline in:
Labour conditions, such as job‐related health
and safety
Qualitative value of employment, such as
whether the jobs resulting from the project
activity
are highly or poorly qualified, temporary or
permanent.
Livelihood of the poor refers to changes
compared to the baseline in:
Poverty alleviation, e.g. changes in living
standards, number of people living under the
poverty line
Access to health care services (hospitals,
doctors, medication, nurses etc.), affordability
of services, reliability and quality of services,
and diseases prevention and treatment,
including HIV AIDS, measles, TB, malaria,
cholera and others.
Access to sanitation including access to
toilets/washrooms. Waste management
facilities that offer the possibility of deposing
Certificates
Children immunized
against measles
Maternal mortality
ratio HIV prevalence
among pregnant
women
Condom use rate of the
contraceptive
prevalence rate
Condom use rate for
high‐risk people
Population with
comprehensive correct
knowledge of

waste in a sanitary way.
Access to an appropriate quantity, quality
and variety of food that is a prerequisite for
health.
Changes in proneness to natural disasters
that may be climate change related (e.g.
droughts, flooding, storms, locust swarms,
Etc.) or unrelated (e.g. earthquakes, volcano
outbreaks)
Long‐term changes that differ from natural
disasters in the sense that they occur
steadily/increasingly but not suddenly (e.g.
community’s dependency on river water from
a river with diminishing volumes of water)
Changes must be directly related to the service
and not an unintended impact.
23
HIV/AIDS/other
diseases
Prevalence and death
rates associated with
malaria
Population rate in
malaria‐risk areas using
effective malaria
prevention and
treatment measures
Prevalence and death
rates associated with
tuberculosis
Proportion of
tuberculosis cases
detected and cured
under directly observed
treatment short course
DOTS (Internationally
recommended TB
control
strategy)
Infant mortality rate
Life expectancy
Number of hospitals
available
Number of doctors
Number of physicians
Number of nurses
Proportion of births
attended by skilled
health personnel
Under‐five mortality
rate
Infant mortality rate
Quality improvement of
health care services
Number of population
with access to
improved sanitation,
urban and rural
Number of population
who can access to
effective waste
management system
Prevalence of
underweight children
under‐five years of age
Proportion of
population below
minimum level of

Access to
affordable and
clean energy
services
Human and
institutional
capacity
Access to energy services refer to changes
compared to the baseline in:
Presence, affordability of services and
reliability of services
Reducing dependency of fuel/ energy imports
that may lead to more sustainable and
affordable energy services in a country.
Also, decrease in risk of political conflicts
caused by energy imports may be included.
Human and institutional capacity refers to
changes compared to the baseline in:
Education & skills: Access to primary,
secondary and tertiary schooling as well as
affordability and quality of education.
Educational activities which are not part of the
usual schooling system, such as environmental
training, awareness raising for health or other
issues, literacy classes for adults, and other
knowledge dissemination.
Gender equality: Livelihood and education for
women that may include special schooling
opportunities as well as other woman‐specific
training, awareness‐raising, etc.
Empowerment. Changes in the social structure,
e.g. caused by a change in the distribution of
income and assets. This may result in shifts in
decision‐making power at project level (e.g.
participation in project executive board,
ownership of CERs etc.), community level (e.g.
community council) or at a higher level.
Especially in communities with diversified
ethnic or religious structures, changes in
income and asset distribution may have an
impact. Especially ownership of CERs or other
direct involvement in the project may support
participation in project decision‐making.
24
dietary energy
consumption
Availability of Reliable
disaster warning and
relief system at
community, local,
regional, and national
levels
Knowledge and
information
dissemination regarding
natural disaster
Energy use
Traditional fuel
consumption
Change in Energy use
Change in Traditional
fuel consumption (% of
total energy
requirements)
Electricity consumption
per capita (kilowatt‐
hours)
Female combined gross
enrolment ratio for
primary, secondary and
tertiary schools
Female Adult literacy
rate
Change in female
earned
income
Change in number of
jobs and positions for
women
Change in decision‐
making structures at
the community, local
government levels
Change in income and
asset distributions by
region, ethnicity,
religion, and socio‐
economic groups
Women in government
or
decision making groups
at community, regional,
ministerial levels

Issues Description Possible parameters
Economic and technological development
Quantitative
Quantitative employment and income Household income
employment
generation refers to changes compared to the generated from the
and income
baseline in:
project
generation
Number of jobs
Income from employment in the formal and
informal sector. Other income, such as from
ownership of CERs, may be included
Balance of
payments and
investment
Technology
transfer and
technological
self‐reliance
Balance of payments and investment refer to
changes compared to the baseline in :
Net foreign currency savings resulting from a
reduction of, for example, fossil fuel imports as
a result of CDM projects.
Investment into a country/region or
technology. Without proper access to
investment, projects may demonstrate
credibility and reliability of loan takers and
trust in the financial structure. Hence future
investments into similar or other activities may
be enabled. Only if financing possibilities are
limited in the country/region or technology, a
positive impact from demonstration of
investment may exist. Investments may come
from national or international sources.
Bilateral and unilateral investment should be
distinguished, since the former do have this
effect of demonstrating the viability of the
host as a destination for investment, whereas
the latter have this to a much lesser extent
Technology transfer and technological self‐
reliance refer to changes compared to the
baseline in:
Technology development as well as adaptation
of new technologies to unproven
circumstances. Technology can be sourced
from outside or inside the country as long as it
is new to this particular region and introduced
in a proven sustainable way. Demonstrating
the viability of technologies new to a
country/region may help in transforming the
energy sector.
Activities that build usable and sustainable
know‐how in a region/country for a
technology, where know‐how was previously
lacking. This capacity building enables spill over
25
Balance of payments
Amount of domestic
investment
Amount of foreign
direct
investment
Number of workshops,
seminars organized,
and training‐related
opportunities held
Number of participants
who attend those
capacity building
activities
R&D Expenditures

9.2 Appendix 2: Examples
effects to the area by replicating similar or
different projects.
Amount of expenditure on technology
between the host and foreign investors
regarding the contribution of domestically
produced equipment, royalty payments and
license fees, imported technical assistance or
the need for subsidies and external technical
support.
Table 7 Guideline on sustainability assessment
These examples serve as a tutorial for energy managers, M&V program managers and proposal
evaluators alike. It should be possible for all three groups to use the same tables, in order to clearly
communicate the aspects to be measured, the indicators for each aspect as well as the specific
parameters chosen. Below are some examples for each of the aspects.
9.2.1 Engineering Aspects
Table 8 is an example on the evaluation of the installation of variable speed drives (VSD) for a water
pumping system. The program evaluation panel determines to evaluate payback time, energy
converting ratio, maintenance plan, optimal pump on/off scheduling plan, and the overall energy
consumption. These evaluation criteria and the corresponding weighting factors are given to the
program manager for program development. Then the program manager follows strictly the
evaluation criteria and develops the corresponding energy efficiency measures to meet these
criteria. For instance, an optimal pump on/off scheduling plan with a short payback period is chosen
together with the VSD installation so as to have a higher score in the evaluation.
Engineering
aspects
Technology Equipment Operation Performance
Installation of VSD Life Energy Maintenance Matching of Energy
cycle converting
system consumption
cost; ratio (from
electrical to
mechanical)
components;
Items to be Payback Pre: 60% Thorough Optimal Pre:
considered while = 3 Post: 90% maintenance scheduling 90MWh/month
scoring for the months
plan
provided to Post:
installation of VSD
match pump
on/off with
water
demand
60MWh/month
Scores (out of 100) 80 70 65 85 75
Weighting factors
(in percentage)
10% 10% 10% 20% 50%
Subtotal score =80*10%+70*10%+65*10%+85*20%+75*50%
Table 8 Example for program evaluation from engineering aspects
26

In the evaluation of a general program, there might be many issues to be considered. Each of these
issues can be listed exactly like the ‘Installation of VSD’ in the above table and evaluated in terms of
T, E, O, and P related considerations. For instance, the following issues can be included in the
evaluation and each of them will be evaluated and scored separately:
Scoping report (including problem statement, planned activities, saving calculations)
Methodologies (including methods for retrofitting, evaluation, baseline, sampling plan,
metering and monitoring)
Certification/calibration procedures
Quality control
Data (from collection, management, to processing)
9.2.2 Environmental Aspects
Table 9 is an example to evaluate the replacement of a diesel generator by a wind turbine system
and a concentrated solar power (CSP) system in a plant from the environment aspects. Note that the
evaluators can further assign separate scores for water and air and the corresponding weighting
factors to calculate the total score for water and air. Energy saving and reduction of SOx emission are
the primary goals of the program. The evaluation panel thus chooses water and SOx as the main
environmental objects to be evaluated. The program manager determines to use the most advanced
technology to minimize energy and water usage, and SOx emission; this includes, for example, the
installation of VSD for all pumps, the recapturing of SOx, etc.
Environmental aspects Technology Equipment Operation Performance
Water
Cooling and Water
Optimal The amount
heat transfer circulating control of water
technology pumps
systems needed daily
involved to
minimize
water usage
installed
Items considered for Evaluation Pump ratings: No baseline to Pre: 400
scoring in water evaluation criterion: Is 50kW (x 5); compare the tons/day;
the
All are VSD control Post: 350
technology driven
system; tons/day
involved most
system
advanced?
developed by
Answer: Yes
a well‐known
ESO
Air (SOx only) SOx emission Equipment to Optimal Amount of
control absorb SOx is in coordination emission
technology place
of SOx re‐ into the air
adopted
processing and
production
line
Items considered for No baseline Absorption rate No baseline to Pre:
scoring in SOx evaluation to compare of 40% is verified compare the 100kg/day
SOx emission,
control Post:
however this
system; 50kg/day
emission
system
control is the
developed by
most efficient
a well‐known
27

Scores (out of 100, for
both water and air)
Weighting factors (in
percentage)
Subtotal score =75*20%+80*15%+85*15%+70*50%
Table 9 Example for program evaluation from environmental aspects
9.2.3 Social Aspects
one ESO
75 80 85 70
20% 15% 15% 50%
Table 10 is an example on the evaluation of the social aspects. The energy manager and the HR
department jointly work together and propose the job creation aspects of an energy efficiency
project. These are elaborated as the number of new jobs (T), specificities about human resources
needs (E), interrelations in the workplace (O), and the ability to perform, given work environment,
resources available (P). An energy evaluation panel could decide on appraisal based on the absolute
new job count (T), the job description, standards, evaluation methods, level of education required,
permanent or temporary position (E), organogram, defined responsibility, communication channels,
develop or improve orientation program for new employees (O), and work environment (P).
Social aspects Technology Equipment Operation Performance
Quality of jobs Number of new
jobs
Items considered in
scoring job quality
Absolute new job
count
Specificities
about human
resource needs
Assessment or
development of
job description,
standards,
evaluation
methods, level
of education
required,
permanent or
temporary
position.
Subtotal score =80*25%+75*35%+60*20%+65*20%
Table 10 Example for program evaluation from social aspects
28
Interrelations
in the
workplace
Organogram,
defined
responsibilities,
communication
channels.
Development
or
improvement
of orientation
program for
new
employees
Scores (out of 100) 80 75 60 65
Weighting factors (in
percentage)
25% 35% 20% 20%
Ability to
perform,
given work
environment,
resources
available.
Assessment
of work
environment

9.2.4 Economic Aspects
Table 11 is an example to evaluate the economic aspects. In this example, it is proposed that the
local contents are evaluated.
Economic aspects Technology Equipment Operation Performance
Local content Higher levels
of locally
manufactured
technology.
Are we using
as much local
content as
possible?
Items which might be
considered in scoring
(e.g. pre‐ and post‐
implementation data and
impact from the program
(savings, etc.))
Scores (out of 100, for
both water and air)
Weighting factors (in
percentage)
Pre‐ and post‐
assessment of
percentage of
imported and
local
technologies
and parts in
plant.
Use of isolated
parts locally
produced.
Increased use
of local content
Subtotal score =75*20%+80*15%+85*15%+70*50%
Table 11 Example for program evaluation from economic aspects
29
Locally
produced parts
are compatible
with imported
ones
Compatibility
and
communication
to local
manufacturers
of needs
75 80 85 70
20% 15% 15% 50%
Identify
possibilities for
local
manufacture
Assessment of
local
manufacturing
opportunities,
identification
of gaps, and
communication
to
manufacturing
plants