Hannu Snellman.pdf - FINSE

FINSE ry symposiums 

“System/Product Lifecycle 

Management” 

Nokia Research Center, , Helsinki, FINLAND, 23rd October 2003 

Continuous assessment of selected 

suppliers using AWARD solution 

Hannu Snellman – Managing Director 

– MiddleWare Oy 

Copyright © Commerce Decisions Ltd 2001-2003 

CD: Magdalen Centre, Oxford Science Park, Oxford, OX4 4GA, UK 

+44 (0)1865 784350 www.commercedecisions.com 

MW: Business Center, Nihtikuja 3A, FIN-02630 Espoo, FINLAND 

+358 (0)9 270 68290 www.middleware.fi

AGENDA 

• Collaborative, efficient 

proposal evaluation 

• Objective frameworks for 

negotiation 

• Continuous assessment of 

suppliers 

• Functionality of AWARD 

solution to buyers and 

bidders 

Copyright © Commerce Decisions Ltd 2001-2003

Introducing Commerce Decisions 

• A technology company completely focussed on supporting buyers 

and bidders on complex procurements 

• Our technology is supporting projects worth over 

$20 billion 

• Head Office in 

Oxford Science Park UK 

• Operations in UK, France 

and USA 

• MiddleWare is Nordic Distributor 

• Extensive usage by Defence, 

Local Government, 

Healthcare, PFI/PPP 

• UK MoD Preferred 

Assessment Tool 


The Market – Contract Lifecycle Management 

Gartner: Largely ignored historically , but worth $20 billion by 2007 

Buy Side 

Create 

Request 

For Proposal 

Conduct Bid 

Assessment 

Select 

Preferred 

Supplier 

Negotiate 

Contract 

Award 

Contract 

Continuous 

Performance 

Assessment 

Acceptance 

Review 

Request for 

Proposal 

Develop Bid 

Document 

Red Team 

Review 

Negotiate 

Contract 

Sign 

Contract 

Deliver 

Against 

Contract 

Hand Over 

To Client 

Sell Side 


The Characteristics of Complex Procurements 

• Low volume, bespoke 

• Multiple selection criteria 

• Multiple stakeholders 

• Huge paper based document 

sets 

• Competition important 

• Intangible products 

• May be product or process 


Achieving Value for Money on Complex Procurements 

• Best Value is important, not just best price 

• To achieve value for money we need to: 

Understand what we mean by value for money in the context of 

a particular procurement 

Measure the value for money offered in supplier proposals 

Optimise value for money through informed and objective 

negotiation 

Validate we are getting value for money as a contract 

progresses 

• Commerce Decisions has helped our customers 

achieve this on complex procurements worth over 

€20 billion. 


Real experiences of proposal evaluation and negotiation 

• No audit trail - indefensible decisions 

• Cost of wrong decisions very high 

• Distributed teams – different departments and locations 

• Costly and time consuming 

• Subjective decisions 

• Difficult to manage information – huge information sets 

• Ad-hoc competition and negotiation processes 

• Lack of process 

• No organisational learning 

• Few supporting tools – reliance on MS office tools 


How can we measure and optimise value for money 

• Need a framework we can use to model and measure 

VfM 

• This framework also needs to provide process 

efficiencies 

• Basic concepts of the frame work 

• Criteria 

• Weighting 

• Measurement 

• Score Types 

• Analysis 

• Time 


What are criteria? 

• Criteria are the means by which we rate the relative 

merits of differing alternatives. 

• Related to requirements, but not the same thing. 

• Independent: 

• colour of the car; 

• acceleration of the car. 

• Complementary: 

• acceleration of the car; 

• power of the engine. 

• Contradictory: 

• power of the engine; 

• fuel efficiency. 


How do we organise criteria? 

• Flat list: 

• difficult to spot gaps, duplication and 

inconsistencies; 

• almost impossible to decide relative 

importance; 

• difficult to summarise and abstract; 

• difficult to manage allocation to 

different people. 

Badge / brand of car 

Colour 

Acceleration 

Power of the engine 

Type of engine 

Fuel efficiency 

Type of fuel 

Saloon / estate / coupe / open-top 

Styling 

Comfort of seating position 

Cruise control 

Electric windows 

Clear instrumentation 

Ease of use of controls 

Gearbox configuration 

Roadholding ability 

Brake feel and efficiency 

Resale values 

Internal noise when driving on motorway 

Passengers airbags 

Electric mirrors 

Climate control 

Side impact protection 

How long has the model been available? 

Financial viability of the manufacturer 

Insurance group 

Tax group 

Feel and fit of interior 

Build quality 

Price 

Cost of routine servicing 

Service interval 

Cost of non-consumable spares 

Warrantee 

Local servicing 

Courtesy car available when in for repair 


What about a hierarchical list? 

• Hierarchical list: 

• similar elements grouped together, so easier to remove duplication 

and inconsistency; 

• logical structure helps identify gaps at each level of abstraction; 

• each layer of the hierarchy is a different level of abstraction; 

• the hierarchy provides a natural mechanism for allocating an area of 

responsibility to an individual. 

Overall 

Financial 

Technical 

Other 

Purchase 

cost 

Running 

cost 

Depreciation 

Performance Safety Comfort 

Make 

Local 

servicing 

Fuel Servicing Other 

Seats 

Climate Noise Space 

Warrantee Insurance Tax 

Front 

Back 


Weighting 

• We weight the criteria relative to each other. 

• It doesn’t matter what they add up to. 

• We can choose numbers and scales that 

are convenient to us. 

• However, in order to aggregate the 

weights up the tree we need to normalise them. 

• We simply re-calculate the weights so that the weights on the 

criteria under a single node add up to 100 (or any other number). 

• AWARD does this automatically. 

Front 

Seats 

Back 

6 4 

Comfort 

33 

Seats 

17 20 


30 

Front 

Back 

60 40 


Sensitivity 

• We are now able to calculate the weight that each 

individual criterion has overall. 

• Start at the top of the tree. 

• For each node criterion multiply the weights of all the criteria under 

it by the weight of the node criteria – then divide by 100 (assuming 

the criteria add up to 100). 

• Repeat for each level of the tree. 

• We call the result the “sensitivity” of the criterion. 

• AWARD does this automatically. 

Comfort 

33 

33 

Seats 

17 20 30 

17 20 


30 

Front 

Back 

60 40 

20 

13 


100 

80 

60 

40 

20 

0 

0 2 4 6 8 10 12 14 16 18 20 

100 

80 

60 

40 

20 

0 

0 100 200 300 400 500 600 700 800 

100 

80 

60 

40 

20 

0 

0 2 4 6 8 10 12 14 16 18 20 

£ Thousands 

100 

80 

60 

40 

20 

0 

Lexus Jaguar Citroen Ford Skoda Mercedes Audi BMW 

Measurement functions 

• We map measurement functions to the criteria 

• We can use the same function to measure more than one criteria 

Overall 

Financial 

Technical 

Other 

Purchase 

cost 

Running 

cost 

Depreciation 


Make 

Local 

servicing 


Seats 



Front 

Back 

Desirability of make 

value 

Fuel consumption 

value 

Insurance cost 

value 

Purchase cost 

value 

litres / 100 km 

£ / annum 

Brand 


Score Types 

• For each criterion we record four scores. 

• Face value (what we are being told by the bidder) 

• Likely value (what we think is the most likely outcome) 

• Optimistic case (what we think the best case scenario is) 

• Pessimistic case (what we think the worst case scenario is) 

• The four scores can be the same. 


value 

100 

80 

60 

40 

F 

O 

L 

P 

value 

100 

80 

60 

40 

L 


20 

20 

0 

0 2 4 6 8 10 12 14 16 18 20 22 


0 

0 2 4 6 8 10 12 14 16 18 20 22 



Score aggregation 

Overall score calculated for each company 

Overall 

Financial 

Technical 

Other 

Purchase 

cost 

Running 

cost 

Depreciation 


Make 

Local 

servicing 


Seats 



Front 

Back 

All scores for each company normalised and aggregated 

up the structure to give summary views 


100 

80 

60 

40 

20 

0 

0 2 4 6 8 10 12 14 16 18 20 

Lexus Jaguar Cit roen Ford Skoda Mercedes Audi BMW 

Assessment teams 

• Scorers are organised into teams; 

• a scorer can be a member of more than one team; 

• a team can have just one scorer. 

• Each team assesses a set of criteria. 

Financial 

Technical 

Other 

Purchase 

cost 

Running 

cost 

Depreciation 


Make 

Local 

servicing 


Seats 



Front 

Back 

Desirability of make 

value 

100 

80 


value 

100 

80 

Insurance cost 

value 

Purchase cost 

value 

100 

80 

60 

60 

60 

40 

40 

40 

20 

20 

20 

0 

0 2 4 6 8 10 12 14 16 18 20 


0 

0 100 200 300 400 500 600 700 800 

£ / annum 

£ 

Thousands 

0 

Brand 

Commercial team 

Technical team 


Assessment team leader 

• The assessment team leader has to award their own 

“consensus” score for each criteria allocated to the 

team. The consensus score is based upon the scores 

provided by the team. 

• May be an “average” of the scores recorded by the scorers. 

• The assessment team leader can change the consensus score. 

• They have to record the justification for the consensus score. 

• Assessment team leaders can see all the scoring 

information from the people within their team, but not 

scoring information from other teams. 


Analysis 

• The VfM model results in a collection of scores for the 

tip criteria that can be aggregated to arrive at a set of 

scores for the summary criteria at any levels. 

• Various analysis can be performed on these scores in 

order to answer the following sorts of questions: 

• Which supplier comes out top overall on the likely score? 

• Where will negotiation make the most difference to VfM? 

• To what extent do my scorers agree on the scoring of the various 

bids? 

• Which bids can I reject at this down-select as they cannot catch up 

with the highest scoring bids? 

• How much uncertainty is there amongst the assessors scoring. 


Time 

• There is an important time dimension to the model 

• Value for money can be optimised during negotiation 

• And through continuous assessment on contract 

100% 

90% 

80% 

70% 

Contractor A % current 

Contractor B % current 

60% 

Contractor A Score 

50% 

Contractor A Out Of 

40% 

Contractor B Score 

30% 

Contractor B Out Of 

20% 

10% 

0% 

Jan-03 

Feb-03 

Mar-03 

Apr-03 

May-03 

Jun-03 

Jul-03 

Aug-03 

Sep-03 

Oct-03 

Nov-03 

Dec-03 

Jan-04 

Feb-04 

Mar-04 

Apr-04 


Continuous Assessment: Introduction 

• The Assessment Phase (AP) forms an important part of 

the UK MODs’ CADMID lifecycle. The use of funded 

competitive APs is becoming more widespread 

particularly for major projects. These competitive 

assessment phases afford the MOD customer an 

opportunity to select the preferred bidder for the D&M 

phase based on more than just the proposals 

submitted by the bidders. This process involves the 

ongoing assessment of the bidders throughout the AP 

and may be based on a combination of deliverable 

reviews, soft issues assessment or bidder audits etc. 


AMS - Smart Acquisition throughout UK MoD 

See at: http://www.ams.mod.uk/ams/ 

Copyright © Commerce Decisions Ltd 2001-2003 

AWARD 100% supports Acquisition Management System (AMS)

Continuous assessment during competitive AP 

There are a number of reasons for undertaking 

continuous assessment within the framework of a 

competitive AP. These can be broken down as follows: 

• Maintain competition further into the procurement 

process; 

• Allow for a more informed bidder selection decision; 

• Reduce risk by gaining a better understanding of: 

• Need 

• Requirements 

• Possibilities/practicalities 

• Evolve the system/solution 

• Provide material to support MG approval 


Elements of continuous assessment 1/3 

There are often a number of elements which are 

assessed during a continuous assessment (CA). 

These may include the following: 

• Assessment of deliverables e.g. 

• Documentation – plans, specifications, registers 

• Simulations 

• Trials 

Assess ongoing aspects e.g. 

• Soft issues 

• Processes (development, quality etc often assessed by 

review or audit) 

• Key Performance Indicators (KPIs) 



The information gathered throughout the CA is used to 

evaluate the bidders (see graph below for an example) 

and to provide guidance to them to ensure that their 

final proposals are as good as possible. 

100% 

90% 

80% 

70% 

Contractor A % current 

Contractor B % current 

60% 

50% 

40% 

30% 

Contractor A Score 

Contractor A Out Of 

Contractor B Score 

Contractor B Out Of 

20% 

10% 

0% 

Jan-03 

Feb-03 

Mar-03 

Apr-03 

May-03 

Jun-03 

Jul-03 

Aug-03 

Sep-03 

Oct-03 

Nov-03 

Dec-03 

Jan-04 

Feb-04 

Mar-04 

Apr-04 



The bidder selection decision is made by consideration of a 

combination of the CA results and the assessment of the bidders’ 

final proposals. This can be assessed and represented by the 

evaluation criteria indicated below: 

Proposal eval team 

CA team 

Overall assessment 

Proposals 

CA 

Proposal crit 

Proposal crit 

Soft issues 

Soft issues 

Soft issues 

Del 2 

Del 1 

Relations 

Relations 

Relations 

Subcontract 

Subcontract 

Subcontract 

Crit x 

Crit y 

Crit z 


Implementing Such a Model is Difficult 

• An example: 

250 criteria 

4 scores per criteria 

30 assessors 

Average 5 assessors per score + 

consensus decision maker 

4 bidders 

• Record and justify 24000 scores 

• Make 4000 consensus decisions and justify 

• Get the exactly correct sub-set of 

scorecards to the correct assessors 

• Turn 24000 scores into meaningful reports 

for decision makers 

• Comply with corporate processes 

• All needs completing in a matter of weeks 

• This is extremely difficult with a 

spreadsheet!! 


Tackling the problem requires tools 

AWARD implements the VfM model 

• Criteria 

• Weighting 

• Measurement 

• Score Types 

• Analysis 

• Time 

Are all entered 

up-front 


AWARD then automates the capture of information 

• Assessors are asked the relevant questions in the optimal order 

• Instructions are given 

• Works over the web or off-line 

• Maintains an audit trail of answers and rationales 


AWARD then automates the analysis 

• See the results developing as scores are entered 

• Just-in-time down-selection 


And Automates the analysis of it 

• Highlight Strengths and Weaknesses of each bid 


Commerce Decisions Professional Service 

• AWARD application configuration 

Covering the installation of the application, database 

configuration and tailoring of the tools and services available 

within AWARD. 

• Process configuration 

Covers the creation of a suitable information structure within 

the tool, creation of a scoring process, creation of users, user 

- groups and roles and the setup of suitable data access 

rights. AWARD is supplied pre - configured to support the 

MACE process. 

• Assessment configuration 

Covers the creation of a specific scoring mechanism (criteria, 

weights etc) in line with the scoring process already defined. 


Press Releases: What our customers say 

• PROJECTS USING COMMERCE DECISIONS AWARD BREAK 

US$20 BILLION IN TOTAL VALUE 

• MoD: Software Accelerates Bid Process For New Multi- 

Million Pound Biological Warfare Detection Programme 

• PPP/PFI: REDDITCH BOROUGH COUNCIL VOTES IN 

FAVOUR OF COMMERCE DECISIONS 

• PPP/PFI: Oxford Radcliffe Hospitals NHS Trust selects 

AWARD software to manage the procurement process for 

major £90m PFI project 

• “We have been extremely impressed with this software” 

• “The tool also provides hard evidence to justify the selection of 

preferred contractors and is an excellent source of data to support 

the debriefing of unsuccessful bidders” 

• “We have been able to use AWARD to increase the speed and 

efficiency of the scoring and evaluation of bids” 


Index of MiddleWare CD-ROM 

AWARDbuy Interactive Demo 

1.000+ 

Pages & 

Slides 

of VfM-E

Hannu Snellman.pdf - FINSE

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?