INNOVATIONS IN IMPACT MEASUREMENT

Innovations In Impact Measurement 

Introduction 

1 

INNOVATIONS 

IN IMPACT 

MEASUREMENT 

Lessons using mobile technology 

from Acumen’s Lean Data Initiative 

and Root Capital’s Client-Centric 

Mobile Measurement.


2 

AUTHORS 

ACKNOWLEDGEMENTS 

Tom Adams is Acumen’s Director of Impact and 

is based in London, heading Acumen’s global work 

on Impact. 

Rohit Gawande is based in Nairobi and leads the 

planning and delivery of Acumen’s work on Impact 

across Africa. 

Scott Overdyke supports the design and execution 

of strategic initiatives at Root Capital, including the 

mobile technology pilots now underway in Latin 

America and Africa. He is based in Boston. 

The case studies on Labournet and KZ Noir were 

co-authored respectively with Ben Brockman and 

Daniel Gastfriend of IDInsight. 

This work has benefited from the generous funding 

and intellectual support of several organizations. 

Principal funding for the projects detailed in 

the report was provided by the ANDE Research 

Initiative, which in turn was generously funded 

by the Rockefeller Foundation, Bernard van Leer 

Foundation and the Multilateral Investment Fund. 

Acumen would also like to thank Omidyar Network 

for their continued support to Lean Data and 

similarly Root Capital is also grateful to Ford 

Foundation for their wider support to its Client- 

Centric Mobile Measurement work. Both Acumen 

and Root are especially appreciative to Saurabh 

Lall for his advice and support to our work as 

well as Steve Wright, Rachel Brooks, Grameen 

Foundation India and to various folks at IDinsight 

for the many ideas they have shared which have 

informed our thinking and approaches.


3 

PARTICIPATING ORGANIZATIONS 

Acumen is changing the way the world tackles poverty 

by investing in companies, leaders and ideas. We invest 

patient capital in businesses whose products and services 

are enabling the poor to transform their lives. Founded by 

Jacqueline Novogratz in 2001, Acumen has invested more 

than $88 million in 82 companies across Africa, Latin 

America and South Asia. We are also developing a global 

community of emerging leaders with the knowledge, skills 

and determination to create a more inclusive world. 

This year, Acumen was named one of Fast Company’s 

Top 10 Most Innovative Not-for-Profit Companies. 

Learn more at www.acumen.org and on Twitter @Acumen. 

Root Capital is pioneering finance for high-impact 

agricultural businesses in Africa, Asia and Latin America. 

We lend capital, deliver financial training, and strengthen 

market connections so that businesses aggregating 

hundreds, and often thousands, of smallholder farmers 

can grow rural prosperity. Since our founding in 1999, 

Root Capital has disbursed more than $900 million 

in loans to 580 businesses and improved incomes for 

more than 1.2 million farm households. Root Capital 

was recognized with the 2015 Impact Award for 

Renewable Resources – Agriculture by the Overseas 

Private Investment Corporation, the U.S. government’s 

development finance institution. Learn more at 

www.rootcapital.org and on Twitter @RootCapital.


4 

CONTENTS 

INTRODUCTION 05 

ABOUT THE PROGRAMMES IN THIS PAPER 07 

ACUMEN’S LEAN DATA 08 

ROOT CAPITAL’S MOBILE IMPACT MEASUREMENT 09 

PART 1: LESSONS TO DATE 10 

EFFECTIVENESS OF MOBILE TECH FOR SOCIAL PERFORMANCE MEASUREMENT 12 

GETTING STARTED: SEVEN STEPS 20 

PART 2: CASE STUDIES 25 

LEAN DATA: SOLARNOW 27 

MOBILE IMPACT MEASUREMENT: UNICAFEC 29 

LEAN DATA: LABOURNET 33 

LEAN DATA: KZ NOIR, PILOTING A “LEAN EVALUATION” 36 

LOOKING FORWARD 41


Introduction 

5 

INTRODUCTION 

Impact investing is booming. The sector 

is attracting greater attention and more 

capital than ever, with an estimated 

USD 60 billion under management in 2015. 1 

Yet as impact investing grows, quality data collection on 

social performance remains the exception rather than the 

norm. While nearly all impact investors — 95% 2 — say that 

they measure and report on impact, current practice is, on 

the whole, limited to output measures of scale: number of 

people reached, number of jobs created. 

While this is disappointing, it is also understandable. 

The prevailing wisdom within the sector is that collecting 

data about social performance is burdensome and expensive, 

and some impact investors and social entrepreneurs would 

assert that it is a distraction from the ‘core’ work of building 

a financially sustainable social enterprise. 3 

Practitioners believe this because we’ve allowed ourselves 

to be convinced, incorrectly, that the tools we inherited from 

traditional monitoring and evaluation (M&E) methodologies 

are the only way to gather social performance data. 

This is no longer the case. 

Thanks to the ubiquity of cellphones and dramatic 

improvements in technology – inexpensive text messaging, 

efficient tablet based data collection and technologies like 

interactive voice response – we are in the position to quickly 

and inexpensively gather meaningful data directly from end 

customers of social enterprises. Moreover we can collect 

this data in ways that delight and engage customers and 

provide the information social enterprises need to optimize 

their businesses. Used with care and creativity these tools 

can open up new channels for investors and enterprises 

to cost-effectively ask questions at the core of social 

performance measurement, such as the poverty profile of 

customers and changes in their wellbeing. 

Building on the encouraging results of early pilots 4 , Acumen 

and Root Capital have expanded their adoption of mobile 

technologies to enhance our capacity to collect, analyze, 

and report on field-level data. Each organization has 

independently launched parallel initiatives: Acumen’s Lean 

Data Initiative and Root Capital’s Client-Centric Mobile 

Measurement. 

Whilst distinct – Root Capital has tended to use tablet 

technology to improve the efficacy of in-person surveying, 

whereas Acumen has largely adopted remote surveying 

using mobile phones – both have applied data collection 

innovations to right-size our approaches to understanding 

social and environmental performance for and with our 

investees. In the process, we have discovered how these data 

collection tools can also inform business-oriented decisions. 

Collecting data on social performance opens up a channel 

to communicate with customers, thus also providing 

opportunity to gather consumer feedback, data on customer 

segmentation, and market intelligence. 

This paper focuses on synthesizing the lessons from 

Acumen’s and Root Capital’s experiences. Our aim is to 

describe our two complementary approaches to mobile 

data management, in the hopes that these descriptions will 

both generate feedback from other organizations already 

engaged in similar efforts and be useful to organizations 

with similar goals and challenges. And whilst this paper 

describes the efforts of two impact investors we believe this 

work has implications beyond impact investing, including 

foundations, governments and NGOs. 

1. Global Impact Investing Network’s 

Investor’s Council, 2015. 

http://www.thegiin.org/impactinvesting/need-to-know/#s8 

2. J.P. Morgan and the Global Impact 

Investing Network, Spotlight on the 

Market: The Impact Investor Survey, 

2014. http://www.thegiin.org/binarydata/2014MarketSpotlight.PDF 

3. Keystone Accountability, 

“What Investees Think”, 2013, p11.“ 

4. See the following paper for discussion 

of Acumen’s pilot using Echo Mobile 

http://www.thegiin.org/binary-data/ 

RESOURCE/download_file/000/000/528- 

1.pdf


Introduction 

6 

The paper is divided into two sections. The first section 

discusses our lessons to date covering both technical 

(data quality) and operational (ease of implementation) 

considerations. It also highlights some of the current 

limitations of implementing such measurement initiatives. 

The second section provides a range of case studies that 

bring this work to life. We hope these real-life examples 

will provide encouragement and inspiration for others to 

try these approaches. 

Our biggest finding in rolling out Lean Data and Client- 

Centric Mobile Measurement is that these approaches 

allow us to focus on our original purpose in supporting 

social enterprises. These enterprises exist to make a 

meaningful change in the lives of low-income customers 

as well as suppliers (i.e., smallholder farmers). Lean Data 

and Client-Centric Mobile Measurement put power into 

the hands of these customers, giving them voice to share 

where and how social enterprises are improving their lives. 

In so doing, we can finally see – in close to real-time and 

at a fraction of the cost of traditional research studies – 

the data we need to understand if we are achieving our 

purposes as agents of change. 

A DIFFERENCE BETWEEN IMPACT & 

SOCIAL PERFORMANCE MEASUREMENT 

The term “impact” is a tricky one. It often means 

different things to different people. This is not helpful. 

Within the impact evaluation profession, to state that 

an intervention has “impact” usually requires a high 

degree of certainty of attribution, based on the existence 

of a relevant control group against which to judge a 

counterfactual (i.e. what would have happened anyway 

without the intervention). 

Because of this definition of “impact,” we have found 

it more helpful to use the term “social performance 

measurement.” Lean Data and Client-Centric Mobile 

Measurement are collecting reported data on social 

change to understand trends and patterns. This data 

gives indications of social change, and the data gathered 

should consider a counterfactual, even if imperfectly 

measured. However, in most cases these data gathering 

exercises do not have formal control groups, largely 

because we have found it impractical in the context of 

the social enterprise. 

The decision to structure our data-gathering in this way 

reflects a core principle of prioritizing efficiency and the 

business realities of a fledgling social enterprise, and the 

belief that while the data collected in this way will have 

limitations, this data is nevertheless much more useful 

to inform decisions than sporadic or no data at all. 

Our principle objective is not to know with certainty 

that impact can be attributed to a particular action 

or intervention. Our objective is to collect data with 

an appropriate degree of rigor that gives voice to our 

customers, including a more objective window into their 

experiences of a given product or service, and helps the 

businesses we invest in use this data to keep an eye on 

their social metrics and manage toward ever improving 

levels of social performance. 

To avoid confusion in this report, we use the term 

social performance measurement rather than impact 

measurement as a more accurate description of the 

data we collect and use to assess the social change we 

believe both we and our respective investees make. 

The only exception is in the case study by Acumen of 

KZ Noir, where the methodology in question involves a 

control group. To the best of our knowledge, this is a first 

attempt at a “Lean Evaluation” in the context of a social 

enterprise.


About The Programmes In This Paper 

7 

ABOUT THE PROGRAMMES IN THIS PAPER 

This paper presents results from two initiatives: 

Acumen’s Lean Data and Root Capital’s Client-Centric 

Mobile Measurement. Both programmes received generous 

support from the Aspen Network of Development 

Entrepreneurs (ANDE) through the Mobile Innovations 

grant. The discussion and case studies presented in this 

paper focus predominantly on specific projects funded 

though this grant. However, the report also benefits from 

both Acumen’s and Root Capital’s wider experience and 

expertise in implementing such projects. 

TECHNOLOGY TESTED 

Specifically, the paper draws from fifteen separate data 

collection projects across our respective investees (ten from 

Acumen in Africa and India, and five from Root Capital in 

Latin America). Through these projects, Acumen and Root 

Capital have surveyed more than 7,000 customers using 

four different types of data collection methods across 

nearly ten different technology or service providers. These 

include the following technology platforms: Taroworks, 

Echo Mobile, mSurvey, Laborlink, SAP’s rural sourcing 

platform, iFormBuilder, Lumira, Enketo Smart Paper and 

Open Data Kit. Projects ranged from four weeks to six 

months, depending on the intensity and complexity 

of the engagement.



8 

ACUMEN’S LEAN DATA 

In early 2014, Acumen created the Lean Data initiative. Lean 

Data is the application of lean experimentation principles to 

the collection and use of social performance data. The core 

philosophy behind Lean Data is to build, from the ground up, 

a data collection mindset and methodology that works for 

social enterprises. Inspired by “lean” design principles, Lean 

Data is an approach that involves a shift in mindset away 

from reporting and compliance and toward gathering data 

that drives decisions. Lean Data uses low cost-technology to 

communicate directly with end customers, generating highquality 

data both quickly and efficiently. 

Applying both a new mindset and methodology, Lean Data 

aims to turn the value proposition for collecting social 

performance data on its head. Rather than imposing topdown 

requests for data, we work collaboratively with social 

enterprises to determine how Lean Data can generate 

valuable, decision-centric data that drives both social 

and business performance. We start with asking our 

entrepreneurs one simple question, “What does successful 

social change look like to you?” and work with them 

throughout the collection process to ensure both their and 

our data collection needs are met. 

Because our portfolio is heterogeneous by sector and 

business model, individual theories of change relating 

to social performance can vary starkly by investee. 

As a consequence, each implementation of Lean Data 

involves a different set of questions to answer, metrics to 

gather, technologies to deploy and methodologies to use. 

Nonetheless, the three core building blocks to Lean Data 

remain: 

Lean Design 

We tailor our measurement and collection approach to the 

unique context of each company, utilizing existing companycustomer 

touch points where possible. 

Lean Surveys 

By keeping our surveys focused and tailored to the 

company’s needs, we gather meaningful information on even 

challenging social performance questions without requiring 

much of the customer’s time. 

Lean Tools 

By using technology, typically leveraging mobile 

phones, Lean Data enables our companies to have quick, 

direct communication with customers even in the most 

remote areas.



9 

ROOT CAPITAL’S CLIENT-CENTRIC 

MOBILE MEASUREMENT 

Root Capital’s mobile measurement, like all of our 

monitoring and evaluation research, strives to evaluate 

social performance in ways that create value for researchers 

and research participants alike. 

Too often, data collection for impact evaluations, regardless 

of the intent, feels extractive to the research participants. 

Such evaluations reinforce real and perceived imbalances 

in power and opportunity between the people doing the 

research and the people being studied. In contrast, like many 

practitioners of market-based approaches to development, 

Root Capital has come to see impact evaluations as one 

among many touchpoints in the customer, employee, or 

supplier relationship. 

For lack of a better term, and because our partners in these 

studies are in fact our lending clients, the internal term 

we use for this approach is “client-centric” evaluation. Our 

intent, however, is not to coin a new term or invent a new 

impact methodology. It is simply to honor the rights of 

the agricultural businesses and small-scale farmers that 

participate in our impact evaluations, and find creative 

ways to deliver more value to them. By doing so, we hope to 

significantly increase the value of the research, notably to 

participants, without proportionately increasing the cost. 

Root Capital began conducting social performance studies 

with clients and their affiliated farmers in 2011, and began 

using tablets and mobile survey software for field-based data 

gathering in 2012. Interestingly, a number of clients began 

asking for Root’s assistance in developing similar datagathering 

capabilities to inform their own activities on the 

ground. We have since piloted a variety of different mobile 

technology platforms with more than twenty-five clients in 

Latin America and Africa. 

Today, Root Capital uses Information and Communication 

Technology (ICT) for data gathering and analysis to fulfill 

three objectives: 

1. Capture social and environmental indicators at the 

enterprise and producer level to better understand the 

impact of Root Capital and our agricultural business 

clients on smallholder farmers; 

2. Capture producer and enterprise level information to 

better inform the business decisions of Root Capital 

clients; and 

3. Capture and aggregate data relevant to Root Capital for 

the monitoring and analysis of credit risk and agronomic 

practices. 

This report describes the ‘mobile-enabled’ component of 

Root Capital’s client-centric approach. While the majority of 

our experiences using mobile technology in data gathering 

have been in Africa, the technologies profiled by Root Capital 

in this report are exclusive to our Latin American operations 

and focus largely on tablet-based data gathering for detailed 

processes (e.g., deep dive impact studies, internal farm 

inspections, and monitoring of agronomic practices). These 

examples differ from the Acumen case studies in that they 

require personal interactions with producers, and have been 

intentionally selected to demonstrate use of one particular 

approach applied in a variety of settings. 

For more information about Root Capital’s client-centric 

approach, including general principles, practical tips, and 

case studies, please see our working paper 

“A Client-Centric Approach: Impact Evaluation That Creates 

Value for Participants.” 

CONNECTION TO METRICS 3.0 

The approaches of Lean Data and Client-Centric Mobile 

Measurement share much in common with each other. 

However, we recognize that they are also very much in 

line with a growing number of progressive movements 

to change the way measurement is conducted. In 

particular, our efforts in this paper aim to build on the 

work of “Metrics 3.0” (http://www.ssireview.org/blog/ 

entry/metrics_3.0_a_new_vision_for_shared_metrics), 

which intends to advance the conversation among 

investors and companies from accountability-driven 

measurement to a performance management approach.


Introduction 

10 

PART 1: 

LESSONS 

TO DATE


Part 1: Lessons To Date 

11 

The explosion of creative data collection 

techniques that leverage mobile phones 

has opened new opportunities to 

communicate with and learn from lowincome 

customers. Indeed, following our 

respective experiences described in this 

paper, Acumen and Root Capital feel more 

certain than ever that these data collection 

methods will remain a core element of 

how both organizations gather social 

performance data. 

However, we’ve also learnt that asking 

questions and collecting data via mobile 

phones takes considerable thought and 

attention, reinforcing the importance of 

collectively building our knowledge base in 

this area. Compared with traditional inperson 

surveying, remote, mobile-phone 

based methods to collect social data are in 

their infancy. Even applying tablet-based 

technology, ostensibly a relatively simple 

upgrade of old fashioned pen and paper 

surveys, requires careful integration of new 

technologies and training of new processes. 

Through our projects both Acumen and 

Root Capital are learning about the best 

ways to implement mobile data collection, 

every bit as much from our failures as 

from our successes. And despite bumps 

in the road, both organizations are 

discovering that with thought, practice and 

perseverance these technologies are helping 

to transform our ability to collect and use 

social performance data.



12 

EFFECTIVENESS OF MOBILE TECH FOR 

SOCIAL PERFORMANCE MEASUREMENT 

A first question when considering the technologies and 

approaches described in this paper is, “so how well do they 

work?” And, despite some challenges, the headline answer 

appears to be “encouragingly well”. 

And if you are currently considering mobile technology and 

lean design principles for your own data collection efforts, 

you’re likely reading these cases with two fundamental 

questions in mind: 

Q1. Is the data accurate, representative, 

and useful? 

Q2. What are my options and what do 

they cost? 

In this section we share what we’re learning with respect 

to both of these questions.



13 

Q1. Is the data accurate, representative, 

and useful? 

Garbage in-garbage out. This age-old maxim of research 

is as relevant to our work to measure social performance 

as it is in any other data collection and impact modelling 

exercise. Indeed, the entire exercise of social performance 

data collection is wasted if the data that is collected is not 

accurate, representative, and useful for decision-making 

purposes. 

The methods profiled in this report are not exactly applesto-apples 

comparisons – the technologies tested allow for 

different numbers of questions, types of questions, answered 

in different levels of detail. However, each of the methods 

do produce results that are on the whole accurate, inclusive, 

and useful in the following ways: 

Data Accuracy 

The question we spend most time and energy considering 

is that of accuracy. After all, if we cannot gather reliable, 

quality data the whole exercise is undermined. Our 

experience shows that it is possible to collect data with 

enough accuracy to help social enterprises better understand 

social performance, as well as make more informed business 

decisions. We are experimenting with different methods to 

validate and improve accuracy, and we have summarized 

our main findings below. 

One way to get a sense of accuracy of mobile surveys is to 

back-check questions with a small sub-sample (typically 

5-10%) using an alternative, more-established survey 

method. Large variations in responses suggest something 

may be awry. Based on this approach our data suggests 

that, whilst accuracy rates appear to be in general good, 

accuracy of responses can vary for multiple and sometimes 

unexpected reasons (e.g. the complexity of the question 

being asked, the length of survey, and even the mood of the 

respondent at the moment they receive the survey). 

We don’t yet know enough about all the reasons for such 

variability, but it does suggest that significant care must be 

taken to collect reliable responses, and repeated testing to 

find out what works and what does not is required. 

However, we have learnt that rather than one tool being 

universally more or less accurate, the degree of accuracy is 

usually more dependent on matching the right question to 

the right technology. For example, questions asking about 

relatively static variables, such as family size or occupation, 

are well-suited for SMS as respondents typically do not 

need clarification or further information. On the other 

hand, questions about spending habits are well-suited for 

in-person, tablet or call centre interviews, given they can 

require further probing, explanation or nuance to get a 

full picture. We are also finding evidence that sensitive or 

confidential information may be best collected by remote 

technologies, such as Interactive Voice Response (IVR) or 

SMS: this gives respondents increased anonymity and may 

lead to more accurate responses. 5 

Our observations on variability rates are shown in Table 

1 below, categorized by technology and question type. To 

date, our data suggest that rather than one technology 

being ‘better’ that another, the most important cause of 

variability between answers is question type. Responses to 

static questions, asking about generally stable indicators 

such as household size, land size or education levels are 

pretty robust via simple SMS or IVR. By contrast, what we 

describe as ‘dynamic’ variables - metrics that may have high 

variability over short time periods (e.g. fuel spending, litres 

of milk consumed, daily wages) – can be effectively collected 

by call-centre and in person. And whilst it is by no means 

universally impossible to measure these dynamic variables 

by SMS or IVR, it requires greater care. 

Table 1: Observed Variability rates by collection method. 

Technology 

SMS 

IVR 

Call Centre 

Range 

Static Questions 

3-5% 

13-23% 

5-17% 

Range 

Dynamic Variables 

6-100% 

17-44% 

8-12% 

All data points shown are from individual data collection projects at an 

Acumen portfolio company. A range represents two separate data collection 

projects falling under the same category. 

5. Schober MF, Conrad FG, Antoun C, 

Ehlen P, Fail S, Hupp AL, et al. (2015) 

Precision and Disclosure in Text and 

Voice Interviews on Smartphones. 

PLoS ONE 10(6): e0128337.



14 

Moreover, just because there is variability between survey 

questions taken remotely by comparison to traditionally, it 

does not necessarily follow that the remote survey response 

is less accurate. It is true that in-person surveying allows 

skilled enumerators to weed out incorrect answers (e.g. it’s 

hard to respond incorrectly about the material or size of your 

house if the enumerator is standing in front of it). However, 

surveying in person also brings its own potential biases, and 

the apparent anonymity or increased remoteness of a text 

may elicit a more honest answer in certain instances. 

The introduction of new biases based on different survey 

methods underscores the importance of continuously 

testing questions to discover which metrics are best suited 

for a particular collection method. We also believe that the 

success of asking particularly complex questions remotely 

may ultimately come down to the way a question is phrased. 

Over time, we’ve learned how to incorporate changes in 

question phrasing to yield more accurate results. 

GROUP OR INDIVIDUAL ACCURACY 

Determining “accuracy” of various mobile data collection 

tools is not as straightforward as it seems. In typical 

social science surveys “accuracy” refers to how close an 

observation comes to the truth. Typically there is always 

some amount of error involved with any survey tool 

because we’re dealing with human subjects who tend to 

misreport or misremember data points about their own 

lives – especially when it involves something complicated 

like agricultural production, health or education. 

This becomes more problematic for mobile data collection 

tools because we are testing the accuracy of the survey 

in addition to the tool itself. Therefore, we think about the 

“truth” as the result of a more robust, previously tested 

data collection technique. Accuracy can be considered 

along two dimensions: 

+ Individual-level 

We are confident that, if a respondent reported they 

live in district Y, the respondent actually lives there. 

+ Group-level 

We are confident that, if 50% of our respondents 

reported living in district Y, 50% of the total 

population lives there. 

Some questions require only group-level accuracy, 

while others require individual-level accuracy. 

For example, a firm looking to estimate the average 

satisfaction levels of its customer base needs only worry 

about group-level accuracy. The satisfaction of any given 

customer is not important: what matters is whether 

customers are satisfied or dissatisfied on average as a 

customer-base. By contrast, a firm looking to identify 

individuals living below the poverty line to qualify them 

for additional services requires individual-level accuracy: 

improperly classifying a poor individual as above the 

poverty line will result in a wrong decision. Acumen and 

Root Capital do not see one ‘accuracy type’ as inherently 

better than the other. Rather, we posit that the correct 

accuracy type to report depends on the core question 

a social enterprise is trying to answer. 

The authors would like to acknowledge ID Insight for their valuable 

contribution on this subjct.



15 

Building a representative sample 

Clearly if no one responds to your request for data, you 

won’t get any data, and your ambitions to measure social 

performance will be scuppered before they’ve started. As 

a result, we carefully track response rates to capture a 

sufficiently large, robust and representative sample in our 

data. 

It is relatively straightforward, when conducting in-person 

surveys, to manage response rates and to plan surveying to 

ensure an unbiased sample. People generally respond well to 

being asked politely and professionally to give their opinion. 

By contrast, with remote mobile surveys, it can be harder 

to ensure that a sufficiently large, representative sample 

respond to get high quality, unbiased responses (just think 

how many times you are asked and respond to a survey on 

your phone or online). Low response rates are by no means a 

uniform experience, and several important factors can boost 

the number of people who respond. 

Response rates are tied to customer’s knowledge and 

perception of the company asking the question, and in some 

cases these effects are strong. For example, we’ve seen that 

micro-finance institutions (MFIs) that meet with customers 

every month to discuss their loans can have response rates 

to remote surveys that are regularly above 50 percent and 

frequently reach 80 or 90 percent. 

Of course a low response rate in of itself is not necessarily 

a problem. It may simply mean that you need to send your 

remote surveys to a larger initial population. Given the 

lower costs of remote surveys this is far less of a challenge 

than when administering surveys or requesting feedback 

in person. Where more care is often needed is if there is a 

systematic reason why only some people will respond to 

your surveys – response bias. This is common to many data 

collection projects, and in our experience, even more so for 

remote methods. For example, in an SMS survey, perhaps 

those who respond are particularly excited or agitated, and 

those who do not respond fail because a large proportion 

do not have a phone or can’t use theirs at the time. In such 

instances, response bias becomes a significant factor. 

While this doesn’t have to mean your data collection is 

useless, it does mean that you treat the findings of any 

data collection exercise with more caution (where possible 

factoring the strength and direction of any bias). Where you 

suspect bias you might also consider repeating your survey in 

person or with another mobile tool to see if there are major 

variations in the data you gather. 

Table 2: Representative survey response rates across technology 

type and nature of customer relationship 

Technology 

SMS 

IVR 

Phone Centre 

Nature of customer relationship 

Indirect 

12-15% 

6-9% 

41-42% 

Direct 

35-65% 

N/A 

62-80% 

All data points shown are from individual data collection projects at 

an investee. A range represents two separate data collection projects falling 

under the same category. 

Not surprisingly, companies with more direct relationships 

get higher responses to surveys overall. The conclusions 

around technology type are less definitive. We’ve not yet 

tested IVR enough to draw any firm conclusions but it is 

increasingly clear that both SMS and call Centres can drive 

high response rates from customers.



16 

TIPS AND CHEATS: RESPONSE RATES 

Tips for constructing SMS 

surveys that encourage higher 

response rates: 

Tips for constructing IVR 

surveys to encourage higher 


Tips for constructing call-centre 

surveys to encourage higher 


1. Sensitize customers or let them 

know that a survey is coming, a 

day or so before you send. A little 

anticipation can be effective. 

2. Double check that the literacy 

rate of your customer base is 

high, and that the language you 

use is most appropriate for SMS. 

For instance, many Kenyans 

use English or Kiswahili for 

SMS, rather than their native or 

regional language. 

3. Start with a compelling 

introduction text, clearly 

explaining who the sender is 

and including a statement of 

confidentiality. 

4. Keep the number of questions 

to a maximum of seven or eight. 

We have observed that response 

rates drop off significantly after 

seven or eight questions. 

5. If you have a longer survey, offer 

incentive as compensation (if 

possible with the local network 

carrier), or split the survey into 

two parts and send them over 

two days. 

6. Take time to ensure your 

questions are crystal clear. As 

soon as someone is confused 

they’ll stop responding. Test your 

questions on a small group first 

and if you’re experimenting with 

‘harder’ questions, put them at 

the end after the easier ones. 

1. Try to sensitize customers or 

let them know that a survey is 

coming, and that the company 

would value their response. 

LaborLink typically hands 

out ‘hotline cards’ to their 

respondents beforehand. These 

cards carry a phone number that 

they ask the respondent to call on 

their own time, thus giving more 

agency to the respondent. 

2. Make an in-person connection. 

Voto Mobile has found that IVR 

works best as a supplement to inperson 

interactions, rather than a 

substitute. 

3. Keep it short. Like SMS, IVR 

respondents tend to drop steadily 

with each additional question 

(see LabourNet case study). 

1. Time your survey, we aim for less 

than 7 minutes 

2. Send a text before to warn it is 

coming, and after to ask about 

the experience 

3. Create a compelling introduction, 

clearly stating who the survey is 

from, and why it is important for 

the customer to respond. We find 

communicating that responses 

will be used to improve their 

service or act on specific feedback 

works particularly well. 

4. Select a time of day where your 

customers are highly likely to 

be home and near their phone. 

For instance, many smallholder 

farmers are in the field during 

the mornings and may not have 

mobile network coverage.



17 

RESPONSE BIAS 

An avenue for further research around remote surveying 

is to compare whether the characteristics of those who 

do not answer mobile data collection surveys drastically 

differs from those who are willing and able to be surveyed 

remotely. In the case study on LabourNet below, Acumen 

and ID Insight found early evidence that respondents 

to the mobile surveys may have differed from those 

surveyed in-person. 

The Center for Global Development (CGD) has published 

preliminary work that shows that those who answer IVR 

calls often systematically differ from the population as 

a whole, with respondents being more urban, male, and 

wealthier than the population at large. 6 Their results 

show that with additional analysis, a representative 

sample can be constructed from these phone surveys; 

however, it can require between 2 and 12 times as many 

calls to be made to achieve a representative sample 

similar to an in-person survey. 

6. Leo, Ben. (2015) Do Mobile Phone 

Surveys Work in Poor Countries? 

Working Paper 398: Center for Global 

Development.



18 

Q2. What are my options and what 

do they cost? 

Perhaps the most obvious appeal of mobile based surveying 

is its potential for significant cost savings. Traditional pen 

and paper surveys are often prohibitively expensive for 

social enterprises, especially those that have dispersed or 

remote customers. 

Our experience to date suggests that while costs are 

typically cheapest with SMS and more expensive inperson 

– hardly surprising – the variation in costs can vary 

significantly based on geographical location. This is driven 

by the relative ubiquity of the technology to the market in 

question. For example, in India where call centres are well 

established, costs for call centres can be as cheap as SMS 

based surveys in Kenya (see chart 1). 

Chart 1: Average survey cost by technology type 

Average Full Survey Cost by Technology Type 

(Based on a 10 question survey delivered to 1000 respondents) 

(assuming 10 question surveys delivered to 1,000 respondents) 

In-Person Tablet - Rwanda 

$8,770 

In-Person Tablet - Peru* 

$8,000 

Phone Center - Kenya 

$6,500 

Phone Center - Uganda 

$3,480 

Phone Center - Kenya 

$1,950 

Phone Center - India 

$1,780 

IVR - India 

$2,600 

SMS - Kenya 

$2,360 

$- $1,000 $2,000 $3,000 $4,000 $5,000 $6,000 $7,000 $8,000 $9,000 $10,000 

*Cost represents the set-up and implementation Year 1 of a 4 year survey, where annualized costs are about $4,800.



19 

The analysis above considers the direct out of pocket costs 

associated with using these technologies excluding the 

staff time. In addition, the speed with which such surveys 

can be implemented can also mean a significantly lower 

opportunity cost with respect to time. Many traditional M&E 

projects can take many months and often years to complete. 

The average time for data collection activities featured here 

was typically counted in weeks, including time for planning 

and analysis. Though in-person tablet-based technology is 

more expensive than remote methods, it can be a significant 

time saver because companies benefit from not having to 

hire and train data entry analysts to convert paper forms to 

digital. In addition, tablets improve data quality if done well, 

which can save future cost and time in re-collecting poor 

quality data. 

Notwithstanding that some measures of social performance 

simply take a long time to manifest (e.g. impact of improved 

nutrition) for companies who want to make decisions 

quickly, or may pivot their business model between baseline 

and endline surveys taken years apart, the question of time 

to deliver data may be equally pressing as cost. 

The question of cost is ultimately a question of value: is 

the benefit of a particular mobile tool or process worth the 

costs in terms of licensing fees, time, training, and added 

complexity? Similarly, which tools offer the best ‘bang for 

the buck’ and ensure that results are both accurate and 

actionable?



20 

GETTING STARTED: SEVEN STEPS 

With the considerations of responsiveness, cost and accuracy 

in mind, how might you get started with mobile-based data 

collection? Our first advice is to start small, start with care, 

but do start. Here we lay out a range of practical steps based 

on our learning that we hope encourages you to take the 

plunge. 

A NOTE OF CAUTION: 

CONSIDER WHAT AND WHY, 

BEFORE GETTING TO HOW. 

Before diving into the Seven Steps, we highly 

recommend that you first define what you want to 

measure and why before defining the how. Ask yourself 

what is the fundamental question (or questions) that 

you are trying to answer? What is the bare minimum 

you’d like to know, and what information is needed to 

get there? What will you do with it once you have the 

data you seek? A common mistake we’ve seen is to 

start collecting data without a clear answer to what and 

why from the beginning. If these are clear, the how can 

become your main focus. 

STEP 1: CHOOSE YOUR TOOL 

Given the discussion above regarding the importance 

of context as well as the varying reliability of different 

questions over different technologies, a natural first question 

is “which technology is right for me?” The good news is that 

there is no fixed answer that applies across all surveys and 

organizations. This means that judgement and experience 

is required, however the following questions might be 

instructive in narrowing down the field of opportunities: 

Data gathering: Remote vs In-person 

Consider the following chart when determining whether you 

can primarily engage in remote survey collection. If any of 

the needs listed below the “In-person” category hold true, 

it’s likely you will need to consider tools and technologies 

appropriate for in-person surveys. 

1. In-person 2. Remote 

Number of questions 

Large number of questions necessary 

(>15 questions) 



21 

Once you can determine if you need in-person vs remote 

data collection, there are multiple options available to collect 

data from customers. The below flow-chart is based on our 

experiments with varying types of remote survey collection 

methods. 

It is also worth stressing that you don’t have to use a single 

tool for any data collection exercise. Indeed we’ve also found 

that mixing and matching methods can be highly effective 

especially for complex social performance metrics. For 

example, when trying to measure net crop income, a remote 

survey close to crop harvest can provide good agricultural 

yield and price data, combined with a focus group better 

able to record input costs in the context of poor records and 

high recall bias (in a focus group the collective memory may 

prove more accurate than individual). 

Start 

Here 

Are literacy rates very low? 

No 

Yes 

Do you need to ask 

over 10 questions? 

Do you need qualitative, 

detailed responses? 

No 

Yes 

Yes 

No 

Sensitive questions? 

Yes 

No 

Budget 

Less 

than 

$5k 

More 

than 

$5k 

Go with SMS Go with Phone Go with IVR



22 

Think Local 

The applicability and even availability of technology for 

undertaking mobile data collection will vary depending on 

where you work. For example, despite cataloguing over 80 ICT 

tools in use by African farmers and agricultural enterprises, 

we have seen far fewer tools with resources and interfaces 

translated into Spanish and available in Latin American 

markets. In Kenya, where mobile penetration is high and 

integrated into many parts of life through platforms such as 

mPesa, using SMS for multiple purposes is part of everyday 

life for most. In neighbouring Ethiopia, mobile penetration 

rates are considerably lower, making some of the remote 

technologies potentially less attractive. However, whilst 

national mobile penetration rates are a good guide to 

identifying the sorts of tools you might use, they can be 

misleading. For instance, Acumen predicted that remote, 

Rwandan coffee farmers, many of whom are living in 

extreme poverty, would have low access to mobiles. But after 

completing an in-person tablet survey, we discovered that 

nearly two-thirds of farmers have access to a mobile phone, 

leading us to reconsider SMS. 

Software Selection 

In selecting the right technology, there are any number of 

factors you might consider (see some of the selection criteria 

mentioned in the following cases for examples). For fieldbased 

data collection, you might also consider the following 

elements when selecting your software application: 

+ If you have limited connectivity, find a software that 

allows you to collect data offline, with a big enough local 

server to accommodate the # of surveys you’ll do before 

having access to internet for syncing. 

STEP 2: 

GATHER MOBILE NUMBERS 

(FOR REMOTE SURVEYING) 

While some companies may have a database of mobile 

numbers collected as part of their business model (e.g. this 

is common with micro lenders), it’s likely that you’ll have 

to gather customer mobile phone numbers prior to starting 

your survey. This can seem challenging but there are several 

clever and cost-effective ways to start gathering mobile 

contact information. Any face-to-face customer interaction 

is an obvious place to start (e.g. a company’s sales force), 

but even for B2B companies including those that design 

or manufacture products but don’t sell directly to end 

customers, there are lots of good options: placing a number 

to register your product on your packaging; holding a radio 

campaign to encourage listeners to register their interest 

through SMS; even just handing out flyers. 7 

Even where an existing list of numbers exists, due to 

mobile number change/attrition – often due to customers 

taking advantage of deals from other carriers - it is worth 

periodically refreshing your database and/or checking that 

customers still have the same numbers. We’ve regularly 

found that close to 1 in 10 numbers in company databases 

are incorrect. One potential solution is to give customers 

a “hotline” card with a talk time incentive paid if the user 

updates a company with a new phone number. Ensuring an 

accurate set of phone numbers is especially important where 

panel data or baseline and endline surveys are concerned. 

+ Find a software that allows you to easily customize 

your survey yourself (including, changing the wording, 

structure, and sequence of questions), rather than having 

to rely on an external programmer. 

+ Make sure that the software you choose runs well on the 

tablets you plan to use; while some software programs are 

device-agnostic, some only run on androids or iPads. 

7. These methods may be prone to higher 

response bias, however (see above 

section on Response Bias for tips)



23 

STEP 3: 

DESIGN YOUR SURVEY 

STEP 4: 

PROTOTYPE IN PERSON 

AND ADAPT DESIGN 

Survey design takes practice, but it is also not as hard as 

often thought. Since there are many quality texts that 

describe best-in-class survey design techniques we wouldn’t 

hope to provide any further general guidance here. 8 

However we have discovered some general rules of thumb 

relating to surveys implemented through mobiles: 

+ Less is more: we have found over and over again that 

asking fewer questions, especially when using SMS and 

IVR is best. Beyond 5-7 questions we see steady fall off 

rates. 

+ If you need to ask a lot of questions, rather than doing it 

all in one blast, try asking half your questions one day 

and then ask if people will opt into a second batch the day 

after. 

+ Customers respond well to open ended questions even 

by SMS – they feel ‘listened to’ and the quality of the 

responses is higher than one might typically expect. 

+ Consider which language is most appropriate. This may 

not always be the same as the spoken language. For 

example in parts of Kenya we’ve found people like callcentre 

questions in Swahili and SMS in English. 

+ Include a question that checks for attention, e.g. “reply 4, 

if you are paying attention” (this is especially true if you’ve 

added a financial incentive to answer your survey which 

can prompt some people to get through your survey as 

quickly as possible to access the reward). 

As with all surveying it is essential to test that your 

questions are phrased properly and that they are wellunderstood 

by respondents. Concepts and phrases you think 

are unambiguous may be interpreted differently by others – 

especially over SMS where the text characters are limited, as 

well as tablets or Interactive Voice Response (IVR) where the 

phrasing is fixed. We have also found that, because remote 

methods such as SMS and IVR remove the opportunity for 

survey respondents to ask for clarification to a question, 

testing should always be done in person, either as an inperson 

focus group or with individual respondents. 

Similarly, if you are using a particular hardware or software 

application for the first time (or simply a new survey within 

a known instrument), be sure to test uploading and syncing 

information in different scenarios using different devices. 

The introduction of photos, signatures, video, or even new 

surveys can have unforeseen impacts on data storage and 

ability to sync, so testing (and potentially retesting) is 

crucial. 

STEP 5: 

PREPARE FOR 

IMPLEMENTATION 

Even implementing a simple looking survey remotely is not 

as simple as it seems. For example, for the Progress out 

of Poverty Index 9 ® (PPI) ®, which is one of our favourite 

surveys, we’ve found we need to set aside anywhere between 

half a day and three days of training for enumerators and 

potentially more time for those who have never conducted 

a survey. Despite being only ten questions, the PPI remains 

nuanced. 

8. E.g. Research Methods; 

Graziano and Raulin 

9. The Progress out of Poverty Index (PPI) 

is a 10 question poverty measurement 

tool developed by Grameen 

Foundation and Mark Schreiner: 

http://www.progressoutofpoverty.org/ 

If the survey is to be performed in-person on mobile devices, 

we will typically offer a group training session of 1-3 days in 

which enumerators first review all questions in the survey 

and very quickly begin using the tablet or handheld device 

for the majority of the training. Both Root and Acumen have 

found that enumerators have been able to quickly learn to 

use both the hardware and mobile survey apps, even when 

they have no relevant technological experience.



24 

STEP 6: 

SURVEY 

Let the fun begin! Having undertaken careful planning and 

iteration of your survey design, it’s time for implementation. 

If performing in-person data collection, we have found the 

following activities helpful: 

+ Ensure that all tablets and handhelds are fully charged 

or have enough back-up power, and carry paper forms as 

backup. 

+ Include optional notes sections throughout the survey in 

the case that respondents have additional comments to 

include. 

Surveyors should bring a notebook to take notes and writeout 

open-ended questions instead of typing these into the 

tablet during the survey. Surveyors usually write faster than 

they type on a tablet or smartphone, and with a notebook 

the respondent doesn’t feel ignored or disengaged as the 

surveyor types. 

STEP 7: 

BACK-CHECK 

We recommend that for those new to remote surveying or 

for cases when a new metric / survey is being collected for 

the first time, that you prepare to back-check a proportion 

of responses. A back-check refers to returning – or ‘going 

back’ – to respondents after they’ve been surveyed and 

confirming that their initial response is accurate. Depending 

on your sample size we recommend that you repeat between 

5-10% of your surveys in person to see if there are any 

material differences. Of course you shouldn’t expect all 

answers to be the same, especially where you’re asking for 

subjective views. But at the same time this is a chance to 

highlight inconsistent data, and potentially reconsider how 

you’ve asked questions. Once a survey is more established 

over multiple uses you may consider back-checking less 

frequently. 

INTEGRATION INTO 

COMPANY PROCESSES 

When implementing a new data system it’s easy to 

ignore established company systems. Most companies 

have the capability to generate and collect data from 

existing processing – think of how many times or in 

what ways a social enterprise interacts with their 

customers. Root Capital’s early mobile pilots in Latin 

America were actually a response to client demand. 

These early pilots involved taking the paper-based 

internal inspection surveys mandated by certification 

programs and translating them into mobile surveys 

for completion by enumerators on a tablet computer. 

For enumerators, the only change was moving from 

paper-based forms to tablet-based forms. When 

clients were able to successfully digitize the form, 

collect information, and perform the analysis, Root 

helped support the creation of additional surveys for 

social, environmental, or agronomic data collection. 

Data points such as these not only help streamline 

an operational process – better understanding supply 

chains, target underserved demographics, or evaluate 

sales staff performance – they can also provide a 

window into learning more about customers. 

Of course creating a ‘data-driven culture’ doesn’t 

happen overnight. Though we have successfully 

implemented mobile data collection projects that led 

to better informed decision making, we have found 

that sustaining these practices takes a shift in the 

culture of an enterprise. It is not enough to simply 

introduce a new technological fix or mandate better 

data collection practices across an organization. 

Successful management teams can show the value of 

data to the field staff collecting and entering data to 

get their buy in. 

11. The Progress out of Poverty Index (PPI) 

is a 10 question poverty measurement 

tool developed by Grameen Foundation 

and Mark Schreiner: http://www. 

progressoutofpoverty.org/


Introduction 

25 

PART 2: 

CASE 

STUDIES 

18


Part 2: Case Studies 

26 

This second section highlights four case 

studies, three from Acumen’s Lean Data 

and one from Root Capital’s Client-Centric 

Mobile Measurement. 

Each case study is split into four sections: 

a description of the background, detail 

on how the data was collected including 

technologies used, a snapshot of the actual 

data gathered, and lastly discussion of 

how it helped the social enterprises make 

specific, data-driven decisions relating to 

their business and social performance. 

We hope they help bring the insights from 

the first section to life. 

The examples are drawn from: 

SolarNow, Solar Energy, Uganda. 

SolarNow provides finance to help rural Ugandans 

purchase high-quality solar home systems and 

appliances. Acumen helped SolarNow run phone 

centre surveys to gather customer feedback, 

understand segmentation, and track a range of 

social performance indicators. 

Unicafec, Agriculture, Peru. 

Unicafec is a coffee cooperative in the Peruvian 

Andes. Root Capital helped Unicafec adopt a 

tablet-based internal inspection system, digitizing 

organizational records and creating management 

dashboards to guide decision-making and 

communication with the cooperative membership. 

LabourNet, Vocational Training, India. 

LabourNet is a vocational training company based 

in Bangalore that provides job-training skills to 

informal sector labourers. Acumen implemented 

a customer profiling and segmentation survey 

comparing the performance of IVR, Call Centre and 

SMS. We also aimed to measure social performance 

of wage increases and employability for LabourNet 

trainees. This Lean Data project was implemented 

in partnership with IDinsight, who co-Author the 

case study. 

KZ Noir, Agriculture, Rwanda. 

KZ Noir is a premium coffee aggregator and 

exporter. Using in-person tablet surveys Acumen 

helped establish and evaluate a Premium Sharing 

Program to track and incentivise higher grade coffee 

production. This Lean Data project was implemented 

in partnership with IDinsight, who co-Author the 

case study.



27 

LEAN DATA: SOLARNOW 

Headlines 

+ Lean Data can be integrated into existing 

company processes and resources. 

+ Qualitative data is often as valuable as 

quantitative data – especially concerning 

customer satisfaction. 

+ Social enterprise management teams can 

act quickly on analysis provided through 

Lean Data, creating a rapid feedback loop. 

+ Asking what consumers themselves say 

is meaningful can generate surprising 

results of high relevance to enterprise 

impact and business performance. 

Description 

Willem Nolens, CEO of Ugandan based energy company 

SolarNow, is determined to succeed where others have 

failed. He aims to light one of Africa’s most under-electrified 

geographies, just 5% of rural Ugandans are connected to a 

national grid, which frequently suffers brown and black outs. 

Those not connected have to make do with expensive and 

often dangerous alternatives such as kerosene. 

Achieving these goals will be no mean feat; challenges 

abound, including lack of awareness, low population 

density, and limited capacity to pay amongst his target 

market consumers. Yet Willem saw such challenges as a 

business opportunity to apply the know-how gathered from 

his microfinance background along with innovations in 

solar technology to spread solar home systems across rural 

Uganda. To date they’ve sold systems to more than 8,500 

households. 

Unlike already established solar players who concentrate 

on solar lanterns and small home systems, Solar Now 

sells a larger system that can be upgraded over time. This 

allows customers to start with a few lights and mobile 

chargers, and progress to larger appliances like fans, 

radios, TVs and refrigerators. In order to help customers 

move up this ‘energy ladder’ Solar Now also provides 

financing at affordable rates over extensive periods of 

18-24 months. But despite the company’s success Willem, 

whose business strategy builds on word of mouth, craved 

greater, quality data about their customers. This would 

allow him to understand who was buying their products 

and their experience of solar, allowing him to provide more 

appropriate loans and services that could both widen and 

deepen access to the company’s products.



28 

Detail 

Given the distributed nature of SolarNow’s customers, 

Acumen decided to use a phone centre to conduct customer 

profiling, impact and satisfaction interviews. Though 

SolarNow’s field staff has regular interaction with their 

customers to collect loan payments, we aimed to separate 

this type of data collection from the loan collection staff 

to prevent potential bias. Separation allows SolarNow’s 

customers to feel more comfortable providing any critical 

feedback, which might be less likely to happen if asked by a 

staff member, in-person. 

To better understand customer profiles, Acumen turned 

to the Progress out of Poverty Index (PPI) survey. SolarNow 

would be able to use this information to gauge their 

penetration into rural markets, the majority of which lives 

in poverty, including the effectiveness of their financing 

in reaching the poorest. To assess customer satisfaction, 

Acumen helped the Company create a simple customer 

satisfaction survey, soliciting mainly qualitative feedback 

about the solar home systems and the company’s customer 

service. Lastly we included some short questions on 

household energy expenditure patterns. 

SolarNow already has an active call center, set up to receive 

incoming customer service calls. Acumen trained the call 

center staff to administer the surveys to a random sample 

of SolarNow customers. Given the PPI survey and customer 

satisfaction are relatively simple questionnaires, Lean 

Data was able to save on time and costs of hiring external 

enumerators by using the company’s existing resources and 

systems. 

And it wasn’t just SolarNow that learnt through this 

experience. By listening to customers we learnt that some 

of our preconceptions about the social value that we aimed 

to create through our investment needed refinement. Whilst 

our initial theory of change for investment majored on the 

health effects of switching away from traditional fuels, when 

we asked consumers about their perception of what was 

meaningful to them they majored on cost savings, but also 

to our surprise increased security and the brightness of light. 

Since then we’ve been focusing carefully on what it takes 

to ask, and hear from customers, with respect to their 

own interpretations of “meaningful” impact. Asking about 

meaningfulness requires care but we are increasingly finding 

that asked the right way people are consistent in how they 

report what is meaningful to them and that these answers 

are highly correlated with changes in outcome based 

indicators of wellbeing. 

Data 

SolarNow collected data from over 200 customers over the 

period of two months. The data show that 49% of SolarNow 

customers are likely to be living on less than $2.50 per 

person per day, indicating a strong reach into even the 

poorest rural communities. 

The survey also captured basic information on how 

customers were using the Solar Now system. Almost 

all customers reported an increase in hours of available 

lighting, with the average customer experiencing an 

increase of 2 hours of light per day. The data also show that 

customers are replacing other, dirtier fuels with Solar Now’s 

clean energy – moving from 6 hours of light from non-Solar 

Now sources, to only 1 hour per day. 

Hours of Light 

10 

8 

6 

4 

2 

0 

SolarNow learned that customers generally have high regard 

of SolarNow’s products, but was also surprised to hear some 

customers had experienced problems with faulty parts and 

installation. The company also learned that many customers 

would pay extra for more and varied types of appliances, 

providing feedback and confirmation of the management 

team’s strategy to further expand its product line. 

Decisions 

Hours of Light per Night by Source: 

Hours of Light per Night Now by Source: vs. Before Now Solar vs. Before Now SolarNow 

7.3 

1.0 

Now 

Non-Solar Now sources 

SolarNow 

Following its first experience of Lean Data the company 

has transformed its approach data collection on customer 

profiling and feedback. The Company now repeats the 

customer service surveys designed by Acumen every 

quarter, and intends to track progress over time across 

their distribution network. Using Lean Data, SolarNow will 

eventually be able to collect more detailed and targeted 

information that allows them to better serve and understand 

their customers’ needs, track its effectiveness at serving the 

poorest Ugandans, and gather ever increasing insights into 

the value customers derive from the company’s product. 

6.1 

Before



29 

CLIENT- CENTRIC MOBILE IMPACT 

MEASUREMENT: UNICAFEC 

Headlines 

+ Small, agricultural businesses (including 

cooperatives) are increasingly seeking out 

technology tools and processes that allow 

for improved data gathering, analysis, and 

visualization to inform decision-making. 

+ The integration of a new technology (e.g., 

tablets and a mobile survey platform) can 

actually reduce the complexity of data 

gathering and analysis through automation 

and standardization. 

+ Simple charts and visual depictions of data 

trends are useful in communicating the 

value add of data analysis to cooperative 

leadership and the broader membership. 

Description 

Late in 2014, Root Capital was conducting household surveys 

with members of Unicafec, a coffee cooperative in the 

Peruvian Andes, when the cooperative general manager, 

Alfredo Alarcon, made a simple request. Noting the tablets 

and digital surveys used by Root Capital’s research team, 

the cooperative manager was curious to know if Root 

Capital could teach Unicafec’s own staff how to use similar 

technology when performing annual farm inspection audits 

– a mandatory exercise for all organic certified cooperatives 

in the region. 10 

Despite a significant annual investment of time and energy 

on the part of the cooperative, Root Capital has found 

that these farm inspections are typically perceived as an 

onerous external requirement rather than an opportunity for 

continuous improvement or to generate actionable marketintelligence. 

+ Costs of tablet-based data gathering are 

frontloaded by the acquisition of hardware 

and the initial time spent training, but low 

operating costs going forward make this an 

attractive investment if the program 

is continued over multiple years. 

10. Smallholder cooperatives 

holding organic certifications 

are typically required to 

perform annual, farm-level 

inspections of associated 

producers to track compliance 

with practices. 

The surveys used by auditors 

take quite a bit of time and effort 

ranging from 30-80 questions and 

covering such topics as agronomic 

practices, farm income, and 

general demographic data.



30 

For example, a typical organic inspection survey may 

contain as many as 75 questions, requiring an in-person 

visit by a trained enumerator to each farm, where producers 

are asked about production, agronomic practices, social and 

economic concerns, and finally forecasts for the coming 

year. While all of that information could be extremely 

valuable from a business planning perspective, very little of 

it makes it into the hands of a cooperative’s leadership in an 

aggregated form, much less an actionable set of analyses. 

Alfredo Alarcon saw Root Capital’s tablet-based 

questionnaire as an opportunity to transform an otherwise 

cumbersome inspection process into a powerful engine for 

insight into the coop’s farmers’ practices and perspectives. 

Better understanding of their farm-level conditions might 

help him to improve his targeting of technical assistance 

and internal credit (i.e., loans the cooperative makes to its 

members for farm inputs). At the very least it would enable 

him to effectively capture data in aggregate form to serve 

as a baseline to measure changes in future years. In January 

2015, Root Capital partnered with Unicafec and two other 

Peruvian coffee cooperatives – Sol y Café and Chirinos - to 

put Alfredo’s theory to the test. 

Detail 

For the past four years, Root Capital’s impact teams have 

used tablets and digital survey platforms to administer 

household-level surveys when gathering information about 

farmers’ socioeconomic situation, production practices, 

and the perceived performance of Root Capital’s client 

businesses. As such, the organization was already familiar 

with a number of software options for conducting surveys 

and thus selected a known vendor for the internal inspection 

pilots based on the following criteria: 

1. Quality of survey platform and demonstrated track 

record of success for the provider 

2. Applicability to demonstrated need of cooperatives 

3. Affordability (



31 

Data 

As of the writing of this publication, the final program 

evaluation is still in process and the cost / benefit analysis 

still underway. However, results and observations to date 

include the following: 

+ Improved data quality 

The margin of error for all participating cooperatives was 

less than 1% (even before data cleaning), versus up to 30% 

for those cooperatives using the previous, paper-based 

inspection process. 

+ Time saving 

In less than four hours, each group was able to efficiently 

transfer and spot-check data to a readily accessible 

Excel table. Previously, the process of transferring select 

information from paper to computer required upwards 

of two months for two inspectors, with high variation in 

the quality and quantity of information uploaded. The 

mobile platform allowed for a saving of approximately 

thirty person-days in data inputting, a time-saving that 

helps allay the upfront costs of hardware acquisition and 

software licensing. 

+ Data usefulness 

All participating cooperatives were able to record, 

analyse, and visualize producer-level and enterprise-level 

information – very little of which was previously captured 

(including some advanced features like geo-mapping). The 

visualization component involved simple graphs produced 

via SAP’s free visualization software, Lumira. This step 

was particularly effective at demonstrating important 

trends in the data. 

+ Operational independence: Each group was able to 

digitize a new survey (monitoring agronomic practices at 

the producer level) in less than two hours. After the final 

training workshop, we expect participating cooperatives to 

lead in the design and execution of all further inspections 

with only minimal offsite support from Root Capital. 

+ Scalability of infrastructure 

The digital survey platform can be utilized for various 

types of inspection and data gathering. Administrators 

can also leverage the internal inspections process to add 

other operational and business-oriented questions to the 

farm level surveys. 

+ Producer acceptance: 

Farmers across all three cooperatives were generally 

curious about the tablets and proud of the professionalism 

of the process. 

+ Costs 

The primary direct costs associated with each cooperative 

intervention, beyond cooperatives’ staff time, include (1) 

tablets (~$250 per), (2) software licensing fee (~$1,000/ 

year), (3) consultant fees for inspector training (~5-10 days 

per). For a 400 member cooperative, the average direct 

costs equate to approximately $5,600 in the first year 

(excluding enumerator time, as that was already incurred 

in the previous process) and approximately $1,300 for 

software licensing fees every year thereafter. 

Decisions 

Now that all three cooperatives are able to digitize surveys 

for inspections and monitoring, clean and analyse the data, 

and create simple charts and graphs to identify trends, 

Root Capital’s goal is to continue to build capacity among 

the cooperative leadership to use the data to (1) inform 

internal decision-making, and (2) communicate back to the 

cooperative membership. Beyond the time and cost savings 

identified above, these are perhaps the primary potential 

benefits of the digital surveys and remain as-yet not fully 

captured by the participating cooperatives. Cooperative 

managers have noted the following ways they would like to 

use this data going forward: 

+ Create producer-level projections and segment coffee 

collections by origin and quality. 

+ Assess weaknesses in production related to quality or 

quantity of coffee at the producer level and better target 

technical assistance by need. 

+ Assess social and environmental weaknesses to prioritize 

impact-oriented investments by the cooperative (e.g., how 

to allocate the Fair trade premiums). 

+ Identify financing needs of producers to inform the 

provision of internal credit by the cooperative.



32 

Noting the success of these pilots in Peru and the strong 

demand from other clients in the region, Root Capital is 

now planning to scale up from five Latin American pilots to 

twenty client engagements in the coming year. Additionally, 

we are leveraging this same technology suite for three new 

purposes: 

1. Root Capital is currently using this same technology to 

create modular surveys for the dynamic monitoring of 

agronomic practices at the farm level – particularly for 

farmers engaged in full-scale coffee farm renovation and 

rehabilitation. This should enable greater transparency 

into the agronomic processes of producers and help these 

businesses to quickly identify problems as they arise. As 

producers hit key milestones, additional financing from 

Root Capital is then unlocked. 

2. Secondly, Root Capital is exploring new ways to use 

the internal inspection data to inform our own credit 

underwriting, risk monitoring, and ongoing social and 

environmental performance measurement. We can 

even envision a future in which these small agricultural 

businesses are able to more effectively monitor and 

report on key indicators of interest to some of their larger 

commercial buyers. 

3. Thirdly, we are currently exploring how our clients can 

use mobile technology to create attractive employment 

opportunities for youth. 

DATA VISUALIZATION 

The “ah-ha” moment for many of the cooperative 

managers and inspectors was often linked to the 

simple graphs and charts that helped data-sets come 

alive. Microsoft Excel remains the primary analytical 

tool used by enterprises within Root Capital’s portfolio, 

and so it was important that any new tool for analysis 

or visualization complement that system and use 

similar keystrokes and functionality. For these early 

pilots, Root Capital used SAP’s Lumira software (free, 

online version) to easily and effectively transform 

Excel data tables into striking graphics with no 

additional technical knowledge required. 

These graphs have now become the primary vehicle 

by which the inspection data is analysed by 

cooperative management as well as the cooperative 

membership during the course of each group’s general 

assembly. Each organization can now see a clear 

visualization of production volumes and quality by 

member according to region, elevation, gender of the 

member, plant type, farm size, agronomic practices 

used and more. They can begin prioritizing the stated 

social and environmental needs of the cooperative 

membership and communicate back to each group 

the actual findings of the surveys as a basis for the 

plan forward.



33 

LEAN DATA: LABOURNET 

(CO-AUTHORED BY IDINSIGHT) 

Headlines 

+ Collecting customer profile data through 

IVR technology can influence a social 

enterprise’s customer support services. 

+ We were not yet able to track quality data 

on complex social performance metrics 

such as income or wages through IVR 

Description 

Sunitha speaks with as many as 50 people a day from the 

middle of a call center in Bangalore, India’s technology hub. 

However, she’s not answering customer service complaints 

for Microsoft or FlipKart, India’s upstart rival to Amazon. 

The customers at the other end of Sunitha’s phone calls are 

a collection of migrant construction workers, prospective 

retail sales clerks, and freshly trained beauticians. Sunitha 

works for LabourNet, a rapidly expanding vocational training 

company with a mission of creating sustainable livelihoods 

for India’s vast pool of informal workers. 

The company would like to use the data Sunitha collects to 

understand and improve its business and social performance 

based on trainee feedback and, eventually, to market to 

new clients by showing its track record. In line with these 

goals, Sunitha calls back trainees three and six months post 

training and asks questions about their current employment 

status, wages, and satisfaction levels with LabourNet’s 

training. Sunitha and her colleagues are expected to try 

calling back every one of the people who have been trained 

by LabourNet – now totalling over 100,000 people. 

LabourNet had set up the calls to collect better data on its 

trainees, but the system turned out to be inefficient due 

to over-surveying, and the data collected did not seem 

believable – trainees regularly reported 5x income increases 

post-training, and near-100% satisfaction rates. The company 

talked to Acumen and IDinsight about wanting better data 

on outcomes. LabourNet’s senior management was especially 

interested in understanding changes in real wages and 

employability post-training, which would help them gauge 

their impact and identify areas for strategic improvement 

going forward.



34 

Detail 

Collecting data from LabourNet customers represented a 

significant challenge. LabourNet interacts most with its 

trainees during courses. Following that the trainees, who 

are typically either migrant construction workers or working 

elsewhere in the informal economy, move on to different job 

sites or new cities. 

The team tested two possible solutions to these challenges: 

IVR calls using LaborLink, a product of the California-based 

Good World Solutions, alongside revamping the company’s 

existing call-centre based impact surveys. Specifically we 

edited caller scripts to reduce bias and ‘leading questions’, 

created uniform translated survey scripts for surveyors, 

shifted to Enketo Smart Paper for data entry rather than 

Excel, and instituted a random sampling approach to reduce 

the number of calls needed by 80%. After training, the new 

phone-based survey was administered by just two LabourNet 

staff. To gauge the pros and cons of these two approaches, 

the team followed up in person with 200 respondents (100 

from each method), covering greater depth and allowing 

comparison of survey performance. 

Data 

The surveys asked questions across several categories 

including, wages & employment status before and after 

LabourNet training; poverty levels using Grameen’s Progress 

out of Poverty Index as well as other demographic questions; 

and lastly general customer feedback to determine what 

additional services may be desirable. 

Of the 1,817 respondents contacted by IVR an encouraging 

fifth of these (340), picked up after three attempts and 

answered the first question. However, only 32% of those 

who picked up completed the full survey, leading to a 6% 

completion rate overall. The table below shows the drop off 

rate per question asked. By contrast, phone-based surveys 

showed higher response rates (45%) and considerably higher 

completion rates (91% of answered calls) despite their 

longer duration (15 minutes on average). Across both IVR 

and phone-based surveys, mobile populations (migrant 

construction workers) responded to surveys at much lower 

rates than trainees in other sectors (retail, beauty and 

wellness, etc). 

Survey Question # vs. Question Response Rate 

(Calculated for 19% Survey of respondents Question # vs. who Question started Response survey) Rate 

(Calculated for 19% of respondents who started survery) 

100% 

80% 

60% 

40% 

20% 

0% 

1 2 3 4 5 6 7 8 9 10 

Survey Question Number 

Response and Completion Rates by Technology 

Response and Completion Rates by Technology 

100% 

80% 

60% 

40% 

20% 

0% 

45% 

41% 

19% 6% 

% Answered 1st Question % Completed Entire Survey 

IVR (N= 1,817) 

Phone Calls (N= 788)



35 

Assessing if people responded differently over the phone 

and on IVR vs in-person, we found that data quality 

variability increased as questions became more complex. Yes 

or No questions showed the highest agreement with 

in-person surveys, followed by multiple choice questions. 

% Agreement Between Remote and In-Person 

Data Collection by Technology* 

100% 

80% 

60% 

40% 

20% 

% Agreement Between Remote and In-Person Data Collection 

by Technology* 

0% 

Are you 

currently 

working? 

(Y/N) 

What type 

of ration card 

do you have? 

(Multiple choice, 

4 Options) 

How many 

income earners 

are in your 

house? 

(Continuous Var.) 

How many 

non-income 

earners are 

in your house? 

(Continuous Var.) 

Decisions 

The pilot highlighted both the opportunities and some of 

the challenges with respect to data collection. The results 

provided an opportunity for reflection on the company’s 

data requirements. LabourNet is now working with a 

Bangalore-based consultant to map their data needs against 

the company’s overall strategy. The number, frequency, 

and content of Sunitha’s follow-up phone calls are also 

under consideration as part of this data mapping project, 

which should save the company in terms of her team’s 

time. However, both the Lean Data team and the company 

discovered just how challenging it can be to measure 

informal sector incomes and employability; likely because of 

the irregularity of the employment and earnings patterns. 

This highlights an example of where more survey design and 

prototyping is required before we can confirm that we can 

(or cannot) collect this kind of dynamic data remotely. 

IVR 

Phone Survey 

Data omitted for respondents who indicated “Don’t Know” on remote technology 

*100% agreement indicates no variation between what respondents answered 

in-person vs over 

Data omitted 

a remote 

for respondents 

method 

who indicated "Don't Know" on remote technology 

*100% agreement indicates no variation between what respondents answered in-person vs over a remote 

method 

Findings in this case study support the suggestions in part 1 

that it is important keep remote surveys, simple and short, 

clearly define ambiguous terms, and pilot the questions in 

person first. In this instance both IVR and call-centre calls 

were effectively cold calls, and depending on the company 

relationship we have now often pre-empted phone or IVR 

calls with an SMS to warn that the survey will be coming.



36 

LEAN DATA: KZ NOIR, TESTING A “LEAN 

EVALUATION” (CO-AUTHORED BY IDINSIGHT) 

Headlines 

+ Social enterprises may be able to adapt 

traditional impact evaluation methods 

to make business decisions, while 

maintaining a Lean approach. 

+ Training operations staff to conduct inperson 

surveying using mobile devices 

can be cost-effective compared to hiring 

an external data collection firm. Field 

staff picked up tablet technology very 

quickly. 

+ The act of data collection itself has the 

potential to affect customer relationships 

in positive ways (e.g. by creating 

opportunities for staff and customers 

to interact) and negative ways (e.g. by 

excluding some customers from data 

collection activities and giving the 

impression of selective treatment). 

Description 

Celestin Twagirumuhoza ascends up the side of a hill, 

stepping over bushes and under branches to find a small hut 

atop a hillside farm. Introducing himself as the manager 

of Buliza, a nearby coffee washing station, he asks the 

farmer if she would be willing to participate in a short 

survey. After explaining further and obtaining the farmer’s 

consent, he takes out a Samsung tablet, opens an electronic 

questionnaire, and proceeds to ask about her farming 

practices and her livelihood. 

Celestin works for KZ Noir 11 , which launched in 2011 when 

its parent company, Kaizen Venture Partners, purchased 

and integrated three existing Rwandan coffee cooperatives. 

The company produces high-grade coffee, providing 

smallholder farmers with higher earnings in the process. 

Yet it faces a challenging market environment: of more than 

180 coffee washing stations in Rwanda selling specialty 

coffee, most operate below 50% capacity, and almost 40% 

are not profitable. High costs, competition, quality control 

challenges, and under-resourced supplier farmers all make 

business challenging for premium-grade coffee brands. 

11. KZ Noir is an Investee of both 

Acumen and Root Capital.



37 

To build its competitive advantage and increase its social 

performance, KZ Noir launched a premium-sharing program 

for the 2015 coffee season. The program trains supplier 

farmers to cultivate, harvest and process premium grade 

coffee with the goal of passing back a portion of the price 

premium obtained on the export market to the farmers. 

Harvesting high quality coffee requires farmers to carefully 

pick the ripest coffee cherries and care closely for trees with 

weeding, pruning, pest control and fertilization. 

KZ Noir hoped that the incentive of an end-of-season 

price premium would drive higher quantity and quality of 

production. To the company’s knowledge, this is the only 

program in Rwanda seeking to directly track the quality of 

smallholder farmers’ groups’ coffee yields for the purpose of 

sharing premiums. 12 

Detail 

As KZ Noir built this program, it sought to set up an 

improved data system that could track the volume and 

quality of coffee sold by each farmer group, gather more 

information on its supplier farmers to learn how to better 

support their growing practices, and also importantly 

make an informed decision on the success (or failure) of the 

premium sharing program from both a financial and social 

perspective. 

Given the complexity of the questions at hand, Acumen 

partnered with IDinsight to design a Lean Evaluation - a 

formal evaluation with control group designed to directly 

inform a decision while minimizing cost, time and any 

imposition to the companyAdditionally the evaluation 

needed to be proportionate, as it would not make sense to 

spend more on an evaluation than the potential profits or 

social impact generated by the premium sharing program. 

Based on these considerations, KZ Noir opted for a statistical 

matching approach using a new farmer registration survey 

at baseline, and KZ Noir’s operational data at endline. 

To support the premium sharing program, the proposed 

evaluation KZ Noir needed a system capable of tracking the 

volume of coffee sold by individual farmers, the quality of 

coffee sold by farmers’ groups throughout the season 13 , and 

the household characteristics and farming needs, such as 

access to inputs, of supplier farmers 

To meet these goals, the team launched a farmer registration 

survey to collect data on farmers eligible to join the 

premium sharing program and a control group. It asked 

a series of short questions on household size, assets and 

agronomic practices, creating a profile for each farmer that 

could later be matched to coffee sales data and analysed 

by KZ Noir leadership in an interactive management 

information system. The team adopted EchoMobile which 

allows both in-person tablet surveying as well as SMS-based 

data collection. This allowed KZ Noir to link its registration 

data (by tablet) to additional data points (by SMS) throughout 

the season. 

With some initial training from Acumen, EchoMobile, and 

IDinsight, on data collection practices (avoiding leading 

questions and piloting surveys with an out-of-sample group), 

KZ Noir utilised its own staff to conduct the registration 

survey using mobile tablets during the off-season, 

minimizing labor costs. IDinsight and Acumen then spent a 

week monitoring data collection in the field finding that KZ 

Noir staff quickly learned to use the mobile tablets. Moreover 

although company staff, with limited experience struggled 

to adopt all data management best practices, such as 

conducting random back-checks from day one of the survey, 

to catch mistakes early ,at the time of writing (prior to the 

endline survey) we are confident that the savings (against 

hiring external enumerators) will more than offset any 

issues of data quality. 

12. While there are other organizations 

that share premiums with supplier 

farmers, we do not know of any 

others that trace the quality of coffee 

supplied by specific farmers’ groups 

and calibrate the bonus shared with 

the group to their quality level. 

13. Ideally in this situation the Company 

would want to track individual 

farmers, rather than farmer groups.



38 

EVALUATION OPTIONS FOR THE PREMIUM SHARING PROGRAM 

Evaluation design 

Sources of outcomes data 

Randomized Trial Statistical Matching Pre-post Household surveys Operational data 

(selected method) 

(selected data source) 

Description 

Randomly determine 

which farmers are 

invited to join the 

program and which 

serve in a control group. 

Compare the groups at 

end of season 

Identify farmers 

similar to those invited 

to join the program. 

Compare participating 

farmers to similar nonparticipating 

farmers at 

end of season 

Assess participating 

farmers before and 

after joining the 

premium sharing 

program, without 

comparing them to 

non-participants. 

Extensive surveys 

investigating farmers’ 

agronomic practices, 

expenditures, and 

revenues 

Transactions records 

and coffee quality 

monitoring data 

Value of 

information 

provided 

High. Randomization 

eliminates potential 

sources of bias in 

measuring impact 

Medium. Controls for 

many confounding 

variables influencing 

coffee quality and 

supply 

Low. Controls for few 

confounding variables 

influencing coffee 

quality and supply 

High. Provides 

estimates of farmer 

profits and practices 

Medium. Provides 

data on the volume 

and quality of coffee 

sold. Possibly less 

accurate. 

Cost 

Medium. Requires 

interviewing a 

“treatment” and 

comparison group at 

baseline 

Medium-High. 

Requires interviewing 

a “treatment” and 

comparison group at 

baseline. Larger sample 

size needed to attain 

same statistical power 

as a randomized trial 

Low. Requires 

interviewing only a 

“treatment” group at 

baseliney 

High. Accurately 

tracking farm profits 

and practices requires 

extensive surveys, 

potentially with 

multiple touch points 

throughout a season 

Low. KZ Noir was 

already planning 

on investing in the 

requisite internal data 

systems, as these 

were required for the 

premium sharing 

program to function 

Operational 

burden 

High. Requires 

denying access to 

eligible farmers; risks 

damaging supplier 

relationships 

Low. Requires 

interviewing 

more farmers at 

baseline, but KZ Noir 

already intended to 

interview farmers for 

informational purposes 

Low. No programmatic 

changes or additional 

surveying was required 

beyond what KZ Noir 

already intended 

High. Requires 

supervising extensive 

data collection; 

demands additional 

time from respondent 

farmers 

Low. Requires 

ensuring that data 

systems KZ Noir 

already planned on 

installing function 

properly



39 

EXAMPLES OF QUESTIONS AND RESULTS 

FROM KZ NOIR’S FARMER REGISTRATION SURVEY 

Operational question 

Data source/survey 

questions 

Answers received 

Is KZ Noir reaching extremely poor 

populations? 

“Progress Out of Poverty Index” 

KZ Noir supplier farmers exhibit high levels of 

extreme poverty (59% live below the $1.25 per 

day power purchasing parity poverty line). 

Are there areas of higher poverty that 

KZ Noir should target with additional 

social services? 

“Progress Out of Poverty Index” 

Poverty levels are roughly equal across KZ 

Noir’s washing stations, with only one washing 

station exhibiting significantly higher poverty 

levels than the average. 

Where are farmers receiving agricultural 

inputs from, and do they need additional 

help sourcing inputs? 

+ Did you use manure/chemical fertilizers 

on your trees? 

+ If yes, where do you get manure/fertilizer 

from? 

+ Did you get free pesticides from the National 

Agricultural Export Development Board 

(NAEB) this year? 

+ Did you also purchase pesticides this year? 

Most farmers acquired chemical fertilizer 

(88%) last season, with smaller numbers 

(but still majorities) acquiring pesticides (66%) 

and using manure (64%). 

Most farmers interviewed received chemical 

fertilizer from their coffee washing stations 

(64%), pesticides from NAEB (62%), and manure 

from their own production (60%). These results 

suggest that KZ Noir may be successfully 

promoting fertilizer use and could raise use 

of other inputs by providing them as well. 

Can KZ Noir reach most of its supplier farmers 

via mobile phones? 

+ How many mobile phones does your 

household own? 

+ What are your phone numbers? 

67% of households own at least one mobile 

phone, indicating that KZ Noir can reach large 

numbers of households via phones, but not all. 

Can KZ Noir disburse end-of-season 

premium payments to farmers via bank 

transfers? 

+ Do you have a bank account? 

+ If yes, which bank do you bank with? 

80% of households interviewed owned at 

least one bank account, indicating that bank 

transfers could be a feasible method of postseason 

payment for most farmers. 

Does the premium sharing program raise 

the quantity and quality of coffee sold to 

KZ Noir? 

KZ Noir’s internal coffee purchase and 

quality tracking data 

Season has not yet finished.



40 

Data 

KZ Noir has not yet completed its first purchasing season 

using this system, but the farmer registration survey has 

provided useful baseline information on its supplier farmers. 

It found that KZ Noir farmers are extremely poor, potentially 

one of the poorest customer-bases Acumen has in its 

portfolio: 59% of farmers interviewed live below the $1.25/ 

day poverty line, slightly fewer than the 2011 national rate 

of 63%; 14 and 51% have not completed primary school. On 

the other hand, 80% of interviewed households hold a bank 

account and 67% own at least one mobile phone, indicating 

that KZ Noir can capitalize on high levels of financial 

inclusion and connectivity by for example offering payments 

by bank transfer rather than cash 

Decisions 

KZ Noir plans to use the lessons learned from the baseline 

data to better inform how it could distribute resources and 

services for next year’s growing season. In addition, the 

Lean Data system helps the company better demonstrate 

transparency through their supply chain, an important 

feature for international coffee importers. Much to 

everyone’s surprise KZ Noir also noticed that competitors 

seemed to launch farmer registration surveys of their own 

in response to our survey. These reactions were potentially 

an effort to pull farmers out of KZ Noir’s orbit and draw their 

loyalty elsewhere. These observations serve as a reminder 

that data collection does not exist in a vacuum, but rather 

interacts with a company’s competitive environment just as 

any other operational function does. 

14. http://data.worldbank.org/indicator/SI.POV.DDAY


Introduction 

41 

LOOKING 

FORWARD


Looking Forward 

42 

We’ve had fun using mobile technologies 

to gather data and we’ve learnt a great deal. 

Not only in terms of the new data we’ve 

gathered across our respective investment 

portfolios, but also around how best to use 

these technologies. So what comes next? 

Seeing how these techniques have performed, both Root 

Capital and Acumen aim to expand their use to larger 

proportions of our respective portfolios. We will also expand 

our use of technologies and tools that align with Lean Data 

and Client-Centric Mobile Measurement approaches. For 

example, Acumen is currently prototyping its first data 

collection sensor. 

By deepening and widening our work, we believe that 

our companies will see tremendous value from collecting 

social performance data. They will learn more about their 

customers, about their social impact, and be able to use this 

data to make better decisions and grow their companies. 

In the end, this is our ultimate goal: to enable the collection 

of data that drives both social and financial return. 

Through spreading the use of these techniques and 

developing an increased understanding of our social impact 

sector-wide, we can all begin to address what Jed Emerson 

rightly describes as a metrics myth: 15 a lack of data on social 

impact in our sector despite the stated commitment to 

its collection. 

Consequently, we encourage others to adopt these 

techniques. Whether you’re an investment manager or 

enterprise, if you have or can get mobile-phone numbers 

for your consumers, send out a survey. We urge you to open 

up channels of communication with your customers, ask for 

their feedback, and listen to how they believe a product 

or service has impacted their life. 

As we’ve reiterated throughout this paper, all you need 

to understand your social performance is a mobile phone 

and the will to start. What could be simpler? 

LEARN MORE, TAKE OUR 

LEAN DATA COURSE 

15. Jed Emerson, “The Metrics Myth,” BlendedValue, Feb. 10, 2015. 

http://www.blendedvalue.org/the-metrics-myth/ 

If you’re interest to learn more about Lean Data or the 

principles of social impact measurement in general why 

not take one of our short online courses on these topics. 

The +Acumen platform hosts courses on impact and 

many other key topics relevant to impact investing and 

social enterprise. Visit http://plusacumen.org/courses/ 

social-impact-2/ for a course on measuring impact and 

http://plusacumen.org/courses/lean-data/ for a course 

on Lean Data.


Website 

www.acumen.org 

Twitter 

@Acumen 

Introduction 

Website 

www.rootcapital.org 

Twitter 

@RootCapital 

43

INNOVATIONS IN IMPACT MEASUREMENT

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?