For the last two years, there has been a mystery about the evidence supporting the past favorable assessments of the scope for reducing poverty using microfinance instruments such as the famous Grameen Bank (GB). The chances for many poor people to benefit from access to this form of credit rest, in part, on solving that mystery.
To understand the mystery we need to go back to an influential paper by Mark Pitt and Shahidur Khandker (PK), published in the 1998 volume of the Journal of Political Economy. PK documented research supported by the World Bank—research that came to provide the most cited scholarly evidence yet to support the view that microcredit helps reduce poverty.
The PK paper was based on their analysis of a sample of about 1800 households in 87 villages in Bangladesh—using the GB eligibility criteria based on landholding to identify the program’s impact. Based on their analysis, PK claimed that providing small amounts of credit, especially to women, could help poor families get out of poverty. They found economically and statistically significant gains in household consumption for women borrowing from GB. It is this finding that led Muhammad Yunus, the founder of the GB, to claim often that 5% of GB borrowers escaped poverty per year.
The mystery appeared in a potentially damaging critique of the PK paper by David Roodman and Jonathan Morduch (RM) in a 2009 Working Paper of the Center for Global Development. Strikingly, RM claimed that PK’s main finding could not be replicated. (RM also raise some concerns about a previous paper by Morduch and a paper by Khandker on the same topic; here I focus solely on their critique of the original PK paper, which gets the bulk of RM’s attention and has clearly been the most influential paper in the literature.) Based on their analysis of the same data set used by PK, RM found that GB borrowing actually made women worse off! However, they distance themselves from this disturbing conclusion, preferring instead to question whether anything could be claimed one way or the other. Thus they conclude that “30 years into the microfinance movement we have little solid evidence that it improves the lives of clients in measurable ways.”
In a posting on his microfinance blog, David Roodman lays down the gauntlet in no uncertain terms: “…I think my paper with Jonathan is the academic equivalent not of a citation but an indictment... It is a long document packed with logic and evidence that the flaws are not merely possible but provable in academic court and important enough to generate wrong results.” Indeed, Roodman goes even further to conclude on the basis of his paper with Morduch that “a lot of research published in prestigious journals is wrong.” This is strong stuff indeed.
But we are still left with the mystery. Either PK or RM must be wrong; they simply cannot get different results with the same model and data. (RM made some changes to the PK dataset, but these matter little.) It would be unwise to make too much of the matter until that mystery is solved.
On March 26, Mark Pitt offered a solution. His new paper refutes RM’s main findings and confirms the main result reported by PK, namely that GB borrowing by women significantly increases household consumption.
Pitt makes two main points. First, he questions the appropriateness of the estimation method used by RM in the real-world situations in which there is a positive minimum to borrowing; his simulations using synthetic data suggest that RM’s method performs poorly in this case. It would seem that RM should have known this about their estimator.
Second, Pitt points to inconsistencies between the control variables used by RM and those used in the original PK study. The control variables are naturally key in any non-experimental study, but the changes made by RM—also not evident to their readers—make a big difference to the results.
Based on Pitt’s findings, it looks like RM did not actually test the PK specification, as they claim. They got different results simply because they estimated a different model, and Pitt questions their model on a number of counts.
It will be interesting to hear how Roodman and Morduch respond to Pitt’s rebuttal of their findings. The ball is now on their side of the academic court.
Join the Conversation
Yes it is all very dramatic, isn't it?! Jonathan Morduch and I hope to have a preliminary response available soon. The task is made harder by my need to finish a book manuscript. For now, I can say that we disagree with many specific points in Pitt's response, but we have also learned from him; and that he has helped us improve our replication. And I would emphasize that our central contention was... not about whether the coefficients are right but whether the instruments are valid, i.e., whether impact has been shown. Pitt does not discuss that, but we will continue to.
Read more Read lessAlso, I would like to provide a fuller version of the quote above, because it is less harsh: I think my paper with Jonathan is the academic equivalent not of a citation but an indictment (not to impugn the professionalism of the authors of the original papers). It is a long document packed with logic and evidence that the flaws are not merely possible but provable in academic court and important enough... to generate wrong results.
Read more Read lessDavid, I am sure your book manuscript is important, but resolving this mystery is surely a matter of high priority—not least at this time in which microfinance is facing a crisis. So we all look forward to your reply. But I hope it will not be about the instruments used by Pitt and Khandekar, which was a much studied topic prior to your paper with Morduch. (See my discussion of PK’s identification... strategy in my review of evaluation methods in the Handbook of Development Economics- http://bit.ly/g5hS4O.) The headline issue of your paper—the issue that grabbed much attention, especially amongst critics of Grameen Bank—was your claim that PK’s data did not in fact indicate any consumption gains to GB borrowers—negative gains in fact. This directly refutes PK. Yet, Pitt has clearly gone deeply into the issue, and come back with a proposed solution to the mystery, indicating that the main mistakes are in your paper, and that the original PK findings are robust. That is the issue at hand. Martin
Read more Read lessThe technical issues and academic back and forth are fascinating, and I look forward to Roodman's reply. But I would also love to hear some discussion on how this debate continues to inform our understanding of microfinance, particularly now that results from randomized evaluations are becoming available. Whoever is more technically correct, I am skeptical of any work where the qualitative results... are so clearly dependent on the technical assumptions involved. Thank you.
Read more Read lessI have just posted an initial response to Pitt's critique: http://blogs.cgdev.org/open_book/?p=6185
I'm baffled by Martin's suggestion that the most important question here is whether the changes associated with borrowing are positive or negative rather than whether the changes are causal. How can it possibly matter what the effect is if there is no reason to believe in the proposed causal basis of the effect? If there is to be reflection and discussion of the impact of microcredit than that conversation... should be held primarily around the Banerjee, Duflo, Glennerster et. al. Spandana study, the Banerjee, Duflo Al-Amana study and the Karlan, Zinman Philippines study which are much more likely to provide reliable evidence of causality, and not just around the PK vs RM debate.
Read more Read lessYou surely can’t contend that only randomized trials can be believed? That would seem to be more a claim of faith than science. Yes, randomization can sometimes be a useful tool. But it can hardly claim to be the only rigorous tool for evaluation. Observational studies will continue to have an important role, especially in the (many) situations in which randomization is not feasible, or it generates... results of doubtful (internal or external) validity. This is a well-rehearsed issue; see my paper, “Should the Randomistas Rule?” in the Feb 2009 issue of the Economists’ Voice. Here is the link: http://bit.ly/dGLX9S. Whatever identification strategy is employed, the claims made should follow from the data and assumptions. In this case, two prominent studies, ostensibly using the same data and methods, have come to radically different conclusions on an important issue. That is the mystery. It turns out that one of the studies got it wrong, as explained in Roodman's post cited above. Of course, we all make mistakes. But I only wish the authors had been a bit more humble about their findings, and checked more carefully before going public; it would not seem to have been too hard for them to have seen their errors. That is a lesson for us all, including the randomistas.
Read more Read lessSee my comment here- http://blogs.cgdev.org/open_book/2011/03/response-to-pitts-response-to-roodman-and-morduchs-replication-of-etc.php#comment-5644
Indeed I'm not claiming that only randomized trials can be believed. But where they are possible and have been done, they should play a large role in the discussion of causality. It no longer makes sense to discuss PK and RM without reference to a large body of evidence that has come along since. I find the vast majority of the arguments against the use of RCTs to be attacking strawman positions that... randomistas don't make, or to be equally applicable to all forms of evidence. But that's a side point. The real question here is on what we should be focusing on in this particular debate. To paraphrase from your comments, Martin, I would put it this way: I only wish the authors (Pitt and Khandker) had been more humble about their findings of causality and made their data and programming public when they announced their findings. It would not seem to have been too hard for them to have seen the errors in their causal claims and prevented misunderstandings and misinterpretations. That is a lesson for us all, including randomistas and observationists.
Read more Read lessSorry if you wanted me to write about something else, but the issue I was raising in my post above was NOT identification. That is not to say that I think identification is unimportant; far from it! And I did at least refer readers to other things I have written on the identifying assumptions in studies of microfinance, including the seminal study by PK. But since you are so keen to make identification... the issue, let me emphasize that I am not arguing against the use of RCTs per se (and I do use them myself at times). I do, however, question whether RCTs deserve the status that their recent followers in development economics have given them, particularly when we face so many large knowledge gaps that cannot be addressed with RCTs. Important and interesting questions should drive our development research, not methodological preferences.
Read more Read less