Syndicate content

In the long run...

Markus Goldstein's picture

Following up on Michael’s post yesterday, I wanted to add a couple of thoughts.

Some of the more impressive results I have seen have come out of papers which take a long term view.   Three examples of this are Esther Duflo’s work on school construction in Indonesia (ungated version here), Hoddinott, et. al’s paper on an early childhood nutrition intervention in Guatemala, and Baird, et. al’s paper on the long term effects of deworming in Kenya. All of these papers help us understand the significant effects of human capital investment on earnings – a critical issue for development policy.    Heck, Baird et. al. even cite Arthur Lewis. 

All of these papers use interventions from childhood and map these effects into adulthood, thereby giving a big push to focusing on early-in-life interventions.   This kind of work would also be key in understanding interventions which might take a long time to manifest (as Michael pointed out) and those which address a cumulative problem. For example, a couple of weeks ago I was talking with a doctor in rural Kenya who claimed that a lot of the older women he was seeing were dying from long-term exposure to smoke from their cooking fires. He noted that this was indeed a problem for children, but there was a second bump for older women.   If we wanted to see the effect of improved cookstoves on (older) adult mortality from pulmonary problems, we would need an evaluation with a significantly longer term horizon than usual.  

So why don’t we see more of this kind of work?  

The most obvious reason is that it takes a long time. Tenure clocks don’t run that long, and grants definitely don’t run that long. One way around this is to go look for an intervention in the past which let’s you identify an effect as Esther’s paper on Indonesia did (and from a policy perspective, please look for a program intervention – a quick look at the literature out there in this vein shows far more on negative shocks – war, famines – than on program effects). Another way around this is to look at more short term effects/variables, for example schooling in the case of deworming. 

Of course, the effects might not be linear over time.   Michael’s point about monitoring is key here, but we should go further and consider multiple follow up surveys – perhaps with quick qualitative work or light survey rounds to tell us when things are getting more interesting and thus that it’s time for a more heavy survey round.   The usefulness of this approach was brought home to me by a recent presentation by Sebastian Galiani where he was talking about watching formalized property rights revert back (at least partially) to informality because of the transaction costs of transferring ownership. 

And then you have the problem of the treated control group. If short term results show a significant benefit, and funds are available, there is a good ethical argument for providing the program to the control group as well.   This is analogous to the “stopping rule” in clinical trials (although in economics we do not approach this with any of the organization including pre-program discussion that they do).    This means that 10 years later you have early beneficiaries and later beneficiaries.   All is not lost – fruitful estimation can be done with these two groups, as is done in the Baird, et. al. paper – and maybe we should be doing more of this.   Moreover, if the treatment applies say to a certain age group, one can also get traction by using cohorts as Esther has done. 

Then there’s the problem of finding the original sample.   If you’ve been doing some interim work you have a better shot, but for longer follow-ups attrition can be high – for example in the Hoddinott, et. al. study, they showed up 25 years later and faced 40 percent attrition. Here’s one sin I have committed on more than one survey – not setting up pretty detailed tracking information in the baseline.   Asking questions like who else in the village knows your family, getting alternate phone numbers including phone numbers of friends and families, getting a GPS reading – these are all things that can be used in a longer term follow up.   The problem is, of course, that you’re not planning on the 15-20 year follow up now.   Think of it as a gift for your children.  

So these are some of the reasons. Other thoughts?   Other ideas on what we might do?  

Comments

Submitted by Thomas on
Nice post Markus! Since you mainly discuss the incentives of researchers, I would like to follow up with some thoughts regarding the incentives of policy makers. I guess that the lack of attention for long-term impacts of development programs is also related to the contracts that are typical for the relationship between donors and "Southern organisations". Typically, Southern organisations have to demonstrate results after 4 or 5 years. After that time period contracts typically end and there might thus not be sufficient incentive left to demonstrate the long-run impact of development programs. Although my knowledge mainly concerns the relationship between Northern NGOs and Southern NGOs I think these contractual terms are rather typical for many relationships between donors and beneficiaries and they might dampen incentives to demonstrate long-run impacts. It might thus be interesting to experiment with different contractual terms to stimulate research concerned with the long-run impact of development programs. I am curious what your thoughts are about this subject.

Thanks Thomas. This is indeed a tricky issue -- given that donors also face a political cycle and hence can be constrained by this as well. Foundations and the NGOs they fund should be able to take a longer view, so it would be interesting to see if any of these pairings might be up for trying something more long term. And yes changing the contracting might be interesting. In addition, some donors are moving to funding based on some metric of results -- so this may change things but we will have to see if they will pay for longer term results...

Submitted by Michael Woolcock on
On other longer-run evaluations, Beth King and Jere Behrman have a nice paper in the World Bank Research Observer (2009, Vol 24, No. 1) on some results in education.