Change vs. Impact

How can changes in the lives of microfinance clients best be measured? How does that differ from tracking impact? Last month I participated in a panel that gathered three MFI practitioners, one social investor, and one researcher from Innovations for Poverty Action (IPA) to find answers to that question.

Anticipating some sparring between the “camps,” I sat between those proposing monitoring tools and non-experimental research as a way to track change, and those supporting impact studies with randomized controlled trials (RCTs) as the way to attribute impact, for example, from a loan or savings product.

Tracking or studying change and measuring impact are fundamentally different things, and there’s a growing schism in microfinance between these two camps. On the one hand, there is a movement composed of social investors, MFIs, management consultants, and researchers that support monitoring and change studies. For this group monitoring indicators, such as those offered by IRIS or MIX, are sufficient: research sans randomization may give a “good enough” orientation of what microfinance really does for clients. This group also supports monitoring outputs, making diagnostics on how processes affect outputs, and non-invasive academic research that does not need control groups and helps managers to make decisions. They want rigor: but the approach is pragmatic, rather than purist. A list of the various monitoring tools can be found at the Social Impact practice of McKinsey.

The impact supporters, on the other hand, say that impact evaluation with RCTs is all that will actually measure and answer the impact question: there’s simply no other way. And a significant amount of money and brainpower is going into it. There are currently over 300 RCTs completed or ongoing in the research portfolios of the main RCT researchers: the World Bank , J-Pal, or IPA. “Tell me what you exactly want to know and we will measure it and give you answers,” the IPA researcher told the practitioners.

At the heart of this discussion are different interpretations of what impact means and radically different drivers.

The one thing we all agree on is that marketing change or monitoring and evaluation as impact studies is misleading, and simply wrong. The very name “change studies”—coined by an MFI—may reflect a growing awareness and understanding of what monitoring can—and can’t—do, which is good news. But the gulf between the practitioner and the research worlds needs bridging if we’re really going to make progress on such an important discussion.