Syndicate content

Of report downloads and impact: Tales of a trending working paper

Merrell Tuck-Primdahl's picture

A recent Policy Research Working Paper “Which World Bank Reports are Widely Read?” garnered a flurry of online coverage over the past week and a half.

Unfortunately, several reports misunderstood the paper’s conclusions. As fellow blogger David Evans pointed out yesterday, a few Tweets and stories implied that most Bank reports are not being used at all. That’s clearly not true. In fact, people across the world downloaded World Bank reports millions of times over the past two years. It is true, however, that certain technical, country-specific, and sector-specific reports (the only ones studied by the working paper) are much less widely read, or were not downloaded via the Documents and Reports database that the authors analyzed. Even if not downloaded, these reports were certainly delivered to the clients who commissioned them, and were often emailed to others, or disseminated the old-fashioned way by printing and hand distribution -- a common practice in many parts of the developing world where we work and where internet access is limited.

Given the scope of the World Bank’s knowledge assets, this is a complicated story. The Bank produces a wide variety of knowledge products, including annual global flagship reports (the World Development Report, the Global Monitoring Report, Global Economic Prospects, and Doing Business);  independent evaluations; books on a wide range of development topics; periodic macroeconomic economic updates at the country, sub-regional and regional levels; comprehensive development databases such as the World Development Indicators; tools for poverty measurement and applied economic analysis such as PovcalNet and ADePT; journals  such as the World Bank Economic Review; policy research working papers; annual reports; knowledge notes, and; specialized economic sector work (ESW) and technical assistance (TA) reports.  

Altogether, our knowledge content received more than 3.4 million downloads during the past two years alone. Yet, the paper’s authors, Doerte Doemeland and James Trevino found that, while analysis had been done on the impact of Bank Group flagship reports and the influence of Bank-authored research papers, much less was known about the influence and reach of ESW and TA – the last group of papers on the list above. ESW and TA are often designed to meet a specific country client’s demands as part of operational work. Doemeland and Trevino focus specifically on the determinants of downloads and citations for ESW and TA between July 2008 and June 2012.

ESW and TA reports are the nitty-gritty of the World Bank's engagement with client countries, and most reach their (sometimes narrow) objectives. Doemeland and Trevino cite a survey which finds that the majority of government counterparts think that ESW and TA reports are effective at achieving their objectives, and more than three quarters of them have actually led to changes in policies, regulations, or institutions as mentioned in the text.

We know these reports are read by the clients who commissioned them and have an impact. But are ESW and TA reports having an impact beyond specific experts in country governments (say, with the education ministry in the case of a report on how to retain girls in middle school in a particular country)? Are they also of interest to experts in other countries, and can we package and share the results to ensure they are used beyond their initial narrow purpose? This is precisely the sort of question being discussed during the World Bank Group’s ongoing change process. For instance, our Global Practices and Cross-cutting Solution Areas will be operational July 1. Their purpose is to share development solutions across boundaries and eliminate sector and regional silos that too often leave the valuable knowledge contained in ESW reports unread by all but a few experts at the country or regional level.

While knowledge sharing takes place in many different settings—through seminars, presentations, blogs, and so on—the authors looked only at downloads and citations to gauge the use of these reports. Actually, they found that some “core” ESW and TA reports that covered multiple sectors in middle-income countries with large populations were extensively downloaded. Meanwhile, others that focused on specific technical questions received much less attention. In fact, the authors found that 31 percent of ESW and TA reports in their sample from the Documents and Reports Database (the only comprehensive database that was available when they prepared the report) were never downloaded.

To some extent, this was to be expected. Several of the ESW and TA reports Doemeland and Trevino studied were made public under the Bank’s Access to Information Policy and were never originally intended for wide dissemination.  Others were designed to inform the Bank’s lending policies or summarize advice on very specific issues.  
 
But their paper also found that reports with the stated objective of informing the public debate received significantly more downloads.  
 
Our hope is that this kind of analysis can push us to find innovative ways to increase our impact and spread the Bank’s knowledge. Doemeland and Trevino discuss how more dissemination efforts may drive greater awareness, and ultimately greater downloads. For example, one author of a well-read ESW report told them how he and his team printed thousands of copies of their country policy report and placed them in libraries across the country.   That’s laudable, but more actions are needed if we are to raise our game.
 
About four years ago the Bank launched an Open Agenda, whereby the institution now shares, free of charge, virtually all our research technical reports. As part of that, an Open Knowledge Repository, was established. This repository has significantly boosted discoverability and dissemination of all Bank reports and has had more that 3 million document downloads over the past two years.   Currently about 3,500 of our ESW and TA reports are on the Open Knowledge Repository; just over 10,000 are on the Documents and Reports database.  

It’s true that, in many cases, we need to become far better at getting this work into the hands of a wider set of practitioners, citizens, entrepreneurs, civil society experts and other development actors. We know our reports are already having an impact, but we should accelerate sharing of detailed knowledge across countries and sectors so that Bank expertise can be tailored and replicated in other countries, where needed.

We want knowledge to be absorbed and re-used across the World Bank Group, as well as by our development partners across the world. Such a solutions-oriented approach means more innovative outreach and dissemination.  This is a big priority across the World Bank Group and is one of the issues we're taking on as part of our broader change process.
 

Comments

Submitted by Flavie Halais on

Merrell - I agree with you. The impact of World Bank reports cannot be measured by the number of downloads. But I think we'll also agree that you're sitting on a huge pile of knowledge whose potential to impact the greater public is underexploited.
There is a tremendous opportunity for the World Bank to explore new ways to disseminate this knowledge by expanding its content production activities and bypass traditional communication channels.
As a journalist, a large part of my work has been to read these reports and extract information pertaining to the issues I report on. This is a highly time-consuming task that few of us have the luxury to do; I'm convinced there is a better way to get the information out there, which implies having the World Bank produce its own content - provided this is something you'd like to engage in as part of your strategic communications activities.

Let me know if you'd like to take this discussion further. You can reach me at flavie.halais at gmail

Best,
Flavie
http://www.flaviehalais.com

Submitted by Anonymous on

"We know these reports are read by the clients who commissioned them and have an impact." Statements like these should be backed up with evidence, which is the whole point of Doemeland and Trevino's paper. They mentioned the surveys done among clients and point out their flaws, so they are not a sound way to measure impact.
Rather that defending what is clearly a problem, more time should be devoted to discussing solutions. PDF posting is simply not the best way to disseminate knowledge online. The World Bank and others should be investing in finding a better way to ensure its knowledge is not just "available" online but easy to consume. That's the whole point of the paper in question.

Submitted by David Crate on

Please could you post a list of the 530 reports that have never been dowloaded. With a link to the pdf. On the front page of the website. "Undiscovered Treasures" . This would be a way to start to make up for dismal dissemination of these reports. Maybe do this in segments by thematic area or region. Add perhaps also lists of rarely downloaded reports, less than 5 downloads.

Some of these report may yet get hudnreds of dowloads because they contain relevant knowledge. Try it.

There is little sense in continuing to hide this World Bank knowledge, now that it is widely reported to be hidden.

Add new comment