Measuring Changes in Client Lives through Microfinance: Contributions of Different Approaches

In the past two years, the publication of three impact evaluations of microcredit programs in India, the Philippines, and Morocco precipitated a spate of press reports questioning[1] the value of microcredit and whether it had positive outcomes for poor people (Karlan and Zinman 2009 and Duflo and Banerjee 2009 and 2010). The impact evaluations used randomized controlled trials (RCTs), an evaluation methodology that randomly assigns an intervention to a treatment group and withholds it from a control group. Widely used in medical trials and particularly in drug trials, the RCT approach is growing in popularity among academics and evaluation specialists alike in the social sciences. There are now more than 300 RCTs completed or ongoing in sectors such as education, health, governance, finance, and the private sector.[2]

Historically, microfinance institutions (MFIs) have focused resources on monitoring rather than on external evaluations. Monitoring has largely consisted of tracking financial indicators. But in recent years, monitoring has expanded to include the collection and tracking of changes in a set of social dimensions identified and agreed to by industry players in the Social Performance Task Force (www.sptf.org). A set of social indicators is now being submitted to the Microfinance Information eXchange (MIX) by more than 350 MFIs, along with their financial indicators.[3]

The microfinance industry has developed a number of other monitoring and assessment tools to measure elements of social performance such as client poverty levels (see Table 1 for a list of social performance measurement initiatives). Social performance assessments can be a useful tool for MFI management purposes. However, the assessments do not permit MFIs to attribute welfare changes to their programs because these tools do not address the counterfactual; what would have happened if a particular intervention had not been introduced? Nonetheless, social performance monitoring and assessment are helpful in understanding how and where services are delivered. The data collected can be used to inform impact evaluations and to understand how process factors, such as staff training or disbursement mechanisms, influence outcomes.

Both quantitative and qualitative research methods are used for data gathering and analysis to uncover changes in client lives from microfinance programs. Depending on the selected research question, some research methods can be better than others (Table 2). The choice is rarely an either or situation, but rather a combination of methods. Researchers use quantitative techniques when seeking precise measurable changes caused by or correlated to a specific intervention. They use qualitative techniques, such as focus groups or structured interviews, when trying to understand processes that interact together and cannot be untangled, and when establishing direct causality may be difficult.

There are different ways to estimate what would have happened without an intervention, randomizing who gets the treatment and who does not is one of them.[4] Quantitative evaluations not involving the random placement of clients (quasi-experiments) have been used in microfinance for the past 20 years[5] and are mostly donor-funded. As other researchers have noted, these methods have certain weaknesses, mainly related to selection bias (e.g., are clients taking credit better entrepreneurs or are they more motivated in the first place?).[6]

What makes RCTs useful for microfinance?

RCTs are used in impact evaluations to pinpoint causality. The first studies that emerged directly addressed the question of the impact of microfinance by examining possible outcomes such as an increase in income or food consumption levels, improved children’s school attendance, and better health outcomes. But increasingly RCTs are being used to examine product design and test if the product features address the needs of clients.

Studies that experiment with different repayment periods, the use of commitment devices for savings, or the elasticity of interest rates are some examples of how RCTs are used to show how services can be improved (Gine 2006 and Ashraf, Karlan, and Yin 2006). It is in the refinement of products and in the testing of the right mix of services that some researchers believe RCTs can be most valuable for microfinance. While many MFIs primarily rely on qualitative research for product design and testing, for MFIs with sufficient internal data systems, RCTs can be a cost-effective alternative.[7]

RCTs raise some important issues for practitioners as well. Controlling who gets the service, the ex-ante set up, and the cost and length of time it takes to get results make RCTs a significant investment. A long-lasting partnership between the practitioner and the researcher is needed. In general, as with any other form of evaluation, there are tradeoffs (Figure 1).

Notwithstanding the tradeoffs, the renewed interest in evidence-based microfinance is welcomed. Microfinance covers a range of products—savings, insurance, group loans, consumption loans, working capital loans, and housing loans. How clients use the type of microfinance service they select and their behavior will result in different outcomes and impacts on their lives. Here are some examples:

•  Savings may help to build a cushion to confront future shocks, as reflected in a recent RCT (Dupas and Robinson 2002).

•  Microinsurance may help poor clients shift from lower value to higher value crops (Udry et al. 2010).

•  Payment services can help clients access funds from family members in urban or overseas labor markets, as M-PESA is doing in Kenya.

•  When credit is invested in a business, clients are likely to increase working capital or assets for the business (i.e., sewing machines), that can potentially lead to increased business profits (Duflo et al. 2009).

RCTs are now testing many of these assumptions and hypotheses about how microfinance services are used and, therefore, the impact they have on households.

Box 1: Monitoring versus Impact Evaluation—Case Study

A Weight Watchers group intends to lose weight, with the target being a 1 to 2 lb. weight loss per week. For Weight Watchers and for the group’s members, a reduction in weight would be evidence of positive change toward the weight loss goal.

The group can monitor its progress in several ways, namely to:

1. Weigh members regularly

2. Track calorific intake

3. Track weekly hours of exercise compared to a set goal

Was the weight loss due to the diet, due to exercise, or due to being part of a weight-loss support group? To get this kind of information, an impact study is needed that randomly assigns individuals to a treatment and a control group.

More evidence is needed on microfinance benefits

Only three RCTs examining the impact of microcredit have been completed, and the results are limited. To date, the idea that capital constraints is the only issue that microfinance addresses seems to be off the mark. Microcredit appears to be appropriate for clients with a relatively high capacity for risk and the entrepreneurial ability to create a business that will allow them to earn returns to pay back loans. However, not everyone is a microentrepreneur at the bottom of the pyramid—or wants to be. But that does not mean that microfinance—meaning a broad range of services that includes savings, money transfers, and microinsurance, not just credit—is not useful in helping poor people to manage their household cashflow.

Indeed, looking at a broader range of research suggests that the greatest use of microfinance products may, in fact, be in helping households smooth their consumption and helping them better face shocks. Newer RCT studies, especially those that focus on noncredit products (such as microinsurance and microsavings) are revealing fascinating outcomes, showing how product design may enhance impact for specific client segments.

Along with other kinds of research approaches, RCTs are helping to build a broad knowledge base on what exactly microfinance does—and does not—do. More research and evidence on clients can refine our understanding of how to achieve full financial inclusion.

Acknowledgments

Special thanks to Xavi Gine, Nathaneal Goldberg, Jake Kendall, Christoph Kneiding, Kate McKee, Jeanette Thomas, Xavier Reille, and Richard Rosenberg for their valuable contributions.

Bibliography

Ashraf, N., D. Karlan, and W. Yin. 2006. “Tying Odysseus to the Mast: Evidence from a Commitment Savings Product in the Philippines.” Quarterly Journal of Economics.

Banerjee, A., E. Duflo, R. Glennerster, and C. Kinnan. 2009. “The Miracle of Microfinance? Evidence from a Randomized Evaluation.” Mass.: J-PAL and MIT.

Bennett, Drake. 2009. “Small Change, Billions of Dollars and a Nobel Prize Later, It Looks Like ‘Microlending’ Doesn’t Actually Do Much to Fight Poverty.” Boston Globe, 20 September.

Bruhn, Miriam, and I. Love. 2009. “The Economic Impact of Banking the Unbanked: Evidence from Mexico.” World Bank Policy Research Working Paper No. WPS 4981. Washington, D.C.: World Bank.

Chen, Martha, and Donald Snodgrass. 2001. “Managing Resources, Activities, and Risk in Urban India: The Impact of SEWA Bank.” Washington, D.C.: Assessing the Impact of Microenterprise Services (AIMS), Management Systems International.

Collins, Daryl, et al. 2009. Portfolios of the Poor: How the World’s Poor Live on $2 a Day. New Jersey: Princeton University Press.

Dupas, Pascaline, and J. Robinson. 2009. “Savings constraints and Microenterprise Development: Evidence from a Field-experiment in Kenya.” Los Angeles, UCLA.

Galasso, Emanuela, Xavier Gine, and Markus Goldstein. n.d. “Impact Evaluation: Methods and Implementation Issues.” PowerPoint presentation. Washington, D.C.: World Bank.

Gertler, Paul J., et al. 2011. Impact Evaluation in Practice. Washington, D.C.: World Bank.

Gine, Xavier, and D. Karlan. 2006. “Group vs. Individual Liability: A Field Experiment in the Philippines.” Policy Research Working Paper 4008. Washington, D.C.: World Bank.

Goldberg, Nathanael. 2005. “Measuring the Impact of Microfinance: Taking Stock of What We Know.” Washington, D.C.: Grameen Foundation.

Goldberg, Nathanael, and Dean Karlan. 2008. “Impact of Credit: How to Measure Impact, and Improve Operations Too.” New York: The Financial Access Initiative and Innovations for Poverty Action.

Karlan, Dean, and Jonathan Morduch. 2010. “Access to Finance.” In Handbook of Development Economics, vol. 5, ed., Dani Rodrick and M. R. Rosenzweig, 4703–778. Amsterdam: Elsevier.

Khandker, Shahidur R. 1998. “Fighting Poverty with Microcredit: Experience in Bangladesh.” New York: Oxford University Press.

Macfarquhar, Neil. 2010. “Banks Making Big Profits from Tiny Loans,” New York Times, 13 April.

McKenzie, David. 2009. “Impact Assessments in Finance and Private Sector Development: What Have We Learned and What Should We Learn?” World Bank Policy Research Working Paper No. 3807. Washington, D.C.: World Bank.

Naudet, Jean David, and Jocelyne Delarue. 2008. “Fostering Impact Evaluation at Agence Française de Développement: A Process of In-House Appropriation and Capacity-Building.” NONIE (Network of Networks on Impact Evaluation) Working Paper No. 2.

Ravaillon, Martin. 2009. “Should the Randomistas Rule? Another Criticism of RCTs.” The Economists’ Voice 6 (2).

Savedoff, William D., Ruth Levine, and Nancy Birdsall. 2006. “When Will We Ever Learn? Improving Lives through Impact Evaluation.” Washington, D.C.: The Evaluation Gap Working Group, Center for Global Development.

The Economist. 2009. “A Partial Marvel,” 16 July.

Townsend, Robert, and J. Kaboski. 2009. “The Impacts of Credit on Village Economies.” MIT Department of Economics Working Paper No. 09-13. Mass.: MIT.

White, Howard. 2009. “Some Reflections on Current Debates in Impact Evaluation.” Working Paper 1, New Delhi: The International Initiative for Impact Evaluation, April.

World Bank. 2004. “Monitoring and Evaluation, Some Tools, Methods and Approaches.” Washington, D.C.: OED, World Bank.

Web sites

Center for Global Development. David Roodman’s Microfinance Open Book Blog. http://blogs.cgdev.org/open_book/category/about-the-bookoutline

CERISE. Impact and Social Performance. http://www.cerise-microfinance.org/-impact-and-social-perfomance-#outils

CGAP. http://www.cgap.org

Financial Access Initiative. http://financialaccess.org/

IFMR blog. http://www.ifmrblog.com

Innovations for Poverty Action. http://poverty-action.org/

MIX Market. Social Performance Data. http://www.mixmarket.org/social-performance-data

MIX. Social Performance Indicators. http://www.themix.org/social-performance/Indicators

Poverty Action Lab. http://www.povertyactionlab.org/

Social Performance Task Force. http://www.sptf.info

1 See, for example, Bennett (2009), Macfarquhar (2010), and The Economist (2009).

2 World Bank.

3 http://www.themix.org/social-performance/Indicators.

4 Other techniques are pre-post, matching, differences-in-differences, instrumental variables, interrupted time series, and regression discontinuity. See the pros and cons of each in http://www.ifc.org/ifcext/sme.nsf/AttachmentsByTitle/MEImpactEvaluation.pdf/$FILE/MEImpactEvaluation.pdf.

5 http://www.gfusa.org/sites/default/files/Measuring-Impact-of-Microfinance.pdf.

6 See, for example, Karlan and Goldberg (2008).

7 MicroSave market research tools, for example, are based on qualitative methods. These tools help MFIs develop or refine new products and analyze problems such as dropouts or loan defaults or, inter alia, help monitor customer service.

May 2011

All CGAP publications are available on the CGAP Web site at www.cgap.org

CGAP 1818 H Street, NW MSN P3-300 Washington, DC 20433 USA

Tel: 202-473-9594 Fax: 202-522-3744

Email: cgap@worldbank.org

© CGAP, 2011

AUTHORS:

Mayada El-Zoghbi and Meritxell Martinez

Table of Contents

Measuring Changes in Client Lives through Microfinance: Contributions of Different Approaches

What makes RCTs useful for microfinance?

More evidence is needed on microfinance benefits

Acknowledgments

Bibliography

Web sites