Measured Profit is not Welfare or is it? Intriguing evidence from when microfinance clients gave up microfinance

I finally got around to reading an intriguing paper by Banerjee, Duflo and Hornbeck that has been on my reading list for a while. This paper is a nice example of making lemonade out of lemons – they had intended to evaluate a health insurance product that a microfinance organization in India made mandatory for its clients in selected villages. This product turned out (ex-post) to be a dud, and many existing microfinance clients in treatment villages voluntarily gave up their microfinance rather than pay for this insurance that they didn’t want. The interesting claim in the paper is that they did this, despite it resulting in “economically substantial and statistically significant losses” to the business. The authors’ claim is that revealed preference therefore suggests that there must be lots of unmeasured costs of running a business that profits don’t capture.

This paper intrigued me for several reasons. First, many RCTs of giving people microfinance really struggle to find effects on business outcomes (although some studies get results in particular subsamples). So it is a little surprising to hear of strong results from existing clients choosing not to not take microfinance. Second, I am a big fan of profits (despite all the measurement issues) as a measure of the success of business programs – in contrast to either productivity, or to household expenditure or subjective welfare. So thinking more about the problems with my favored measure is useful.

What do they do/find?
They worked with SKS in two districts in Northern Karnataka, India. Villages were randomly assigned into two groups: 101 treatment villages, and 100 control villages. In the treatment villages all new clients and renewing clients were required to purchase a health insurance policy, which provided coverage only for catastrophic events, hospitalization, and maternal care, at a cost of Rs. 525 (relative to a loan renewal size of around Rs. 9,400). This proved unpopular, was eventually made voluntarily, and then discontinued. They have a baseline sample of approximately 29 SKS clients per village, taken in Dec 2006-Mar 2007. They then have an endline sample of these households taken in 2009-2010, approximately two years after the enrolment decisions, with very low attrition (  

• It appears that approximately 40 percent of households owned a business at baseline (sample of 2150 households). However, only 32% of these (683 households) were still operating the business at endline, while 6% of those without businesses at baseline had started a business by endline (310 households).
• Clients in treatment villages were 22 percentage points less likely to take a loan in the year after the treatment, which persisted after the policy was reversed – at the endline households in treated villages were 16 percentage points less likely to take a loan. This reduction is similar in magnitude for the subsample operating a business at endline.
• Treatment has no effect on survival rates, but has negative but not statistically significant point estimates for business outcomes (less asset spending, less workers, less sales, and less profits).
• The authors then say that the low survival rate is an issue, so let’s focus on the group of households with a business at endline (acknowledging this is potentially endogenous, but saying there doesn’t seem to be much selection) – and then they get larger and statistically significant negative impacts on these business outcomes. The LATE estimate is then that losing microfinance as a result of this intervention results in a Rs. 20,000 reduction in annual profit. i.e. people are voluntarily giving up this much in profit to save paying Rs. 525 in insurance fees! However, consumption levels and life satisfaction are unaffected. The authors interpret this as suggesting that there are unmeasured costs of running a business that aren’t being captured by profits.

My thoughts:
This is a thought-provoking paper, and quite provocative for what it means for microfinance. However, I have several queries and doubts:
 

• Measurement of profits as annual profit: the authors use a single follow-up survey to have individuals recall a year of profits. Most studies of microenterprises use a week or a month as the recall period, finding business owners struggle to recall profits over longer horizons. Especially given the high rate of churn in these businesses, you may have people coming in and out of business a lot, and many of them not operating for a full year. Heuristics whereby they scale up the last month to a year will then yield inaccurate results. Of course whether this affects the treatment estimates depends in part on whether such error is correlated with treatment – one can argue either way on this; but if the levels of reported profits are too high, and treatment has a percent increase, then the estimated treatment effect in levels is likely to be overstated too.
• Interest rate elasticity, anchoring etc.: the APR on the loans was 24%, the insurance premium could be bundled into the loan and then represents a 6% increase. So this is a non-trivial increase in the interest rate. It is not surprising that fewer people will take loans when the interest rate increases. Moreover, the response seems like to reflect possible behavioral biases like anchoring – willingness to pay 30% is likely to be different when you were anchored on 24% than if you were just offered 30% in the first place. For example, Hastings and Shapiro look at responses to rising gasoline prices, and find that people behave as if a $2 rise in gas prices costs them tens of thousands of dollars. The result is that revealed preference may not reveal preferences. It is also unclear what the other associated costs of getting the insurance were – e.g. if clients had to fill out a bunch of forms, attend meetings to learn about the change, etc. then these would be additional costs.
• Confidence intervals: A 95% confidence interval for their treatment effect is (-730,-51000). So at the bottom end of the confidence interval the loss is not that different from the cost of the insurance.
• Slicing and dicing the data: I don’t find the exercise of focusing on those who still had businesses remaining at the endline that convincing, especially when two-thirds of the businesses shut down. So let’s just focus on those who had businesses at baseline. The authors’ approach is to put in zeros for profits for those who close down. The LATE may be around a Rs. 6900 (1449/0.21) loss for them (with a confidence interval including zero). We don’t know what fraction of those who closed down had taken a loan and how this differs with treatment status, but it seems likely to be a sizeable share.  I assume they still had to pay back their loan even if they closed the business. The result is that profits would be highly negative for these businesses, rather than zero. They would be more negative for the treatment group due to the higher interest rate, but more negative for the control group due to more of them taking out loans.  This would then further affect the estimated treatment effect.

None of these is damning, but they make the result less than completely convincing to me. But even if we are convinced, should we then view this as saying something is wrong with using profits?  Not necessarily. There is now a large literature on why don’t the poor take supposedly profitable investments (like reinvesting in their firms, applying fertilizer, migrating to cities, etc.). In those papers the assumption is that the investments are profitable, and then all sorts of reasons (market failures, behavioral reasons, information asymmetries, etc.) have been put forward to explain why the profitable investment is not taken. This paper could then be read along those lines (e.g. why don’t the poor take the profitable investment of taking a lousy insurance product in order to retain access to credit?) – which would then have us thinking about all those same factors, rather than saying that profits don’t capture welfare.