The past week has seen the World Bank building covered in banners and messages promoting the release of the 2012 World Development Report and the annual meetings. One of my colleagues drew my attention to this claim of impact on the sidewalk outside the Bank:
Curious as to where this number came from, and with the footpath not surprisingly devoid of footnotes, I tried searching for the source of this claim of impact. The IFC’s twitter feed and facebook page showed up, stating that “Data from IFC clients across all regions shows 2.4 million jobs provided in 2010, of which 665,000 were jobs for women”, with the source being the just released 2011 IFC annual report (which reports that data from 615 investment clients across all regions and industry sectors show that they employed more than 630,000 women in 2010, and that data from clients for which they had three years of data showed a 14% increase in female employment over 3 years). As an example of where this type of number comes from, the report states “In India, IFC invested in a tea production company that provided employment for almost 32,000 people in 2010.”
So this is the first misstatement – the data refer to one part, not the whole of the World Bank Group. Similar numbers appear in the 2010 IFC report which states: “We know that it takes more than volume to meet the needs of the poor. That is why we carefully target our resources, selecting where our financing and advice can be deployed most effectively. And we set measurable goals to gauge our impact, and improve our performance. In 2009, our clients provided 2.2 million jobs, including nearly 514,000 in the manufacturing and services sectors”. This same aggregate statistic is repeated under the headings “Providing value for money”, “Global challenges and impact” and “poverty and unemployment: impact around the world”.
Clearly this is in no way a measure of impact, and presenting it as such is both disappointing to both researchers working on credible measures of the impacts of different projects, as well as to the readers who are being presented this information as if it is in any way informative about the effects of the World Bank Group’s work. I was in a meeting recently where the argument was put forward that for operational needs one need not strive towards rigorous high-quality impact evaluations, but just “good enough” evaluations, where “good enough” did not involve attempts to compare results to a reliable counterfactual. If such numbers can just be presented and not have anybody call them on it, then little wonder that the appetite for doing serious work is low.
So why does this not tell us anything about impact? This is probably a rhetorical question for most readers, but given that such statements are printed on the pavement, it is perhaps worth pointing out some of the issues with it:
- It is obvious that investing in a company does not mean that your impact is the number of employees hired by this company – otherwise we should just invest $1 in each of Walmart, McDonalds, Carrefour and other large multinationals and have an amazing jobs per dollar return.
- It takes no account of what these firms would have hired without the IFC’s investment- that is, there is no comparison to a counterfactual.
- Even if there were 665,000 additional workers hired by these companies, this would not necessarily mean 665,000 new jobs created – many of these workers would not have come from unemployment.
- The numbers the IFC does provide in the 2011 report suggests the growth in female employment in its clients was 14% over three years – an order of magnitude less than suggested by the 366,000 and 665,000 figures, and this is without even considering what growth would have been in the absence of IFC investment.
- The contemporaneous timeframe is unlikely to be an appropriate one for looking at jobs – investments and advisory services like training programs are likely to take some time to have impacts on employment, so most of the job creation in 2010 from the IFC’s work likely comes from projects they carried out in say 2005-2008, not those carried out in the past year.
We’ve previously had a debate on this blog about the relative merits of portfolio evaluation vs single project evaluation. Clearly it would be great from a public communications point of view if we could say something like “World Bank and IFC projects collectively caused X million new jobs to be created in the past 2 years”. However, we are far from such a point, and the second-best solution is surely not to make statements like that in the photograph that tells us nothing about impact. I have much more sympathy (although still many reservations) for the new efforts the IFC is doing to estimate the overall impact of its regulatory reform work by using the evidence from rigorous micro studies of the impacts of reforms in some countries and extrapolating to the overall portfolio.
I believe that the World Bank and the IFC do carry out a lot of projects that offer large benefits to the countries implementing them, and we are starting to build a body of evidence that rigorously shows what these impacts are. In my opinion, we would be far better to highlight these success stories where we actually have evidence than to present aggregate numbers that are meaningless. So how about banners that instead say things like:
· Land titling reform in Rwanda increased female land ownership and tripled the proportion of women who invested in their land through soil conservation.
· A pilot program in Jordan found job vouchers quadrupled the share of married graduates in employment 8 months after graduation (see page 301 in the WDR, ongoing work is measuring the longer-term impacts).
· Cambodia's scholarship program, supported by the World Bank, increased school attendance rates of adolescent girls by about 25 percentage points.
The WDR 2012 contains many such examples – and we would be far better highlighting the results of these serious attempts to measure impact than using aggregate numbers that are not informative.
Join the Conversation