Understanding What Works: Why Qualitative Research Matters

At the recent “Reaching the Poorest Global Meeting 2012” in Paris, I had the privilege of participating in a research panel, alongside Esther Duflo from JPAL and Dean Karlan from IPA. We were tasked to present key findings from our respective evaluations of the CGAP-Ford Graduation pilot programs. While the other panellists shared results from randomized control trials, my role was to share insights from qualitative research across programs in Haiti, India, Pakistan, and Ethiopia.

Why invest in qualitative research, given the rigorous impact results from the RCTs? Taking a long, deep look into the lives of program participants is essential for understanding the nuances behind the RCT results. Simply put, it gives you a more complete picture of what works, what doesn’t – and why: how the process of change unfolded in the lives of participants, why some participants succeed while others did not, and where RCT results show no net impact (as in the case of SKS in India), the qualitative research managed to explain where things went wrong.

To gauge participants’ perspectives on how they felt they progressed in the program, our researchers conducted a ‘staircase’ exercise, where the bottom step represents ‘extreme hardship’, and the top step represents ‘complete security and happiness.’ Participants placed themselves on the staircase in the first interview, and the concluding interview (from six months to two years after the program ended), and articulated their movement. It was clear, that even across country contexts, participants that climbed the furthest along the staircase (whom we refer to as ‘fast climbers’) were those who felt that the program responded to their specific constraints.

In Ethiopia, for instance, participants generally felt that the Productive Safety Net Program (PSNP) was important in meeting their consumption needs, but that they would remain extremely poor without cash income. By providing productive assets and skills to engage in the market, fast climbers felt that REST’s Graduation program responded directly to a long-standing economic constraint.

In the same vein, those who felt that they did not progress along the staircase shared two characteristics:

1. They did not feel that the program addressed their constraints: this is best illustrated by OCT’s program in Pakistan, where the majority of participants were slow climbers suffering from major illnesses, with no access to public healthcare. It was not within the program’s remit to provide treatment for serious ailments, nor did participants’ income from their enterprises increase enough to offset this enormous physical and financial burden.
2. Those who didn’t progress were the most vulnerable among the respondents: some people enter the program on a weaker footing than others. The research has shown us that households that have no male support, more dependents than earners, experience serious health shocks, and psycho-social constraints have a much harder time making use of program inputs. According to a slow climber in India, “My husband drinks, he beats me if I go to meetings, and he has tried to destroy my assets. Tell me, how much can I do on my own?” The study of the Graduation program in SKS, India showed that those who lacked the resources to manage an asset chose to opt out of the program and did not participate.

Qualitative research also sheds light on the issue of sustainability – after the program ends, who is able to sustain gains? The evaluation of Fonkoze’s Graduation program in Haiti illustrated that those who graduate into microfinance typically continue to build their enterprises, savings, and sustain higher incomes. However, those who ‘fall out of the system’ and do not continue on a post-program trajectory, are likely to erode their gains. Interestingly, those who had the support and cooperation of a male partner were more likely to continue building their enterprises and savings, while those who had husbands who drank, gambled, and depleted the household of its resources, found it difficult to maintain their assets once program support withdrew.

Our research showed that asset choice mattered. For instance, those who chose petty trade typically performed the best in the long-term, given that this is the only asset that gave participants a daily source of income, and were least susceptible to external shocks. However, those who chose petty trade were also the ‘risk-takers’ that demonstrated greater confidence from the onset. While most participants resort to choosing livestock, the research has shown that participants have a tendency to ‘collect’ goats. As a participant in Haiti articulated, “having many goats makes us a big person in the village. And if we are saving for a wedding or a big expense, we can sell all the goats in one go for a large sum.” However, a natural disaster or epidemic can wipe out the entire herd of livestock in a sweep, depleting the household of its entire productive asset base. This occurred in India, Pakistan, and Haiti over the course of the research, plunging households back into extreme poverty post-program.

Here are some concluding thoughts on what qualitative research has shown us:

• One cannot generalize that the Graduation program will benefit all the extreme poor equally. Variations exist among the extreme poor: some are better geared for entrepreneurship than others.
• There is an important role for the state. Given that there are slow climbers, and households that deplete their gains in the long-run, having a national social protection strategy is just as important as program implementation.
• Programs need to think long and hard about the context. Is the program designed to respond to the constraints of the population? Will there be take-up of the most lucrative enterprise options? For a program to be successful, it must respond to the poverty traps that participants experience.
• One cannot underestimate the power of ‘coaching’ and household visits, which is the most important element in building self-confidence and an enterprising spirit. For programs that focused on this, the personal transformation even among slow climbers was nothing short of remarkable. In Haiti, over 90% of participants felt that they had ‘a best friend’ since joining the program, referring to their case manager, or weekly coach. A respondent elaborated by saying, “when someone comes to visit you, asks about you, you feel important. Because of my case manager, I feel like I am somebody.”