Syndicate content

Checking up on the assets of the knowledge bank

Adam Wagstaff's picture



Bhanwar Gopal, an artist from the Barefoot College, prepares traditional Rajasthani masks for plays and puppet shows with material from recycled World Bank reports. "We keep getting these reports that no one reads, so we decided to put them to some use," founder Bunker Roy says. [Source and image: BBC]

Regardless of its veracity (we’ll come to that in a moment), the BBC’s story raises a couple of serious questions. Exactly how much does the Bank publish? And does it have any impact?

The second question is, of course, hard to answer. But as Martin Ravallion and I found out when we tried to answer both questions, even the first isn’t easily answered.

The Bank has a large online publications database.  It’s comprehensive in its coverage of Bank-published books and edited volumes: we counted 2,000 books and 500 edited volumes. The database also includes nearly 4,000 working papers. What surprised us is that the database misses almost completely the 9,000 or so journal articles that have been authored by Bank staff (mostly in the fields of economics and development), as well as the nearly 5,000 book chapters (not all of which are in Bank-published volumes). One quick conclusion, then: the Bank’s official publications database actually includes only a minority of Bank publications! The knowledge bank doesn’t keep good records on its assets!

A second conclusion: 20,000 is a lot of publications. Berkeley faculty have written more economics articles than Bank staff. And on one count Harvard faculty have too. But the Bank is ahead of the other 14 top universities we looked at, and is well ahead of the other development agencies, including the IMF. In development, the Bank is ahead of everyone: Bank staff have published twice as many journal articles as faculty at Berkeley and Oxford—the two universities with the largest counts.

So, the Bank does publish a lot. But do its publications have any impact?
In our paper we get at this using citations—not just citations in established academic journals, but also (courtesy of Google Scholar) citations in new journals, online journals, books, dissertations, and technical reports. Citations tell us how far a publication has influenced the way scholars and report-writers think about a topic.

It’s reasonable enough to expect scholars to influence the thinking of scholars and report-writers. But is it a reasonable expectation of the Bank?

We think so. The Bank calls itself the “knowledge bank” and talks of itself as a generator of new knowledge, not just a synthesizer of existing knowledge. Why otherwise would it have a dedicated research department? And why would so many of its staff outside the research department publish so frequently in scholarly journals?

We estimate that nearly 70% of Bank publications have been cited at least once. Journal articles are cited the most. And the citation data for Bank-authored journal articles look good in comparison to universities. In development, the Bank’s ranking among universities is as high on citations as it is on the number of articles.

What about publications other than journal articles? The Bank’s working papers also get cited more than its books, which in turn get cited more often than its book chapters. Worryingly, half of the latter have never been cited at all.

Interestingly, some Bank book series fare better than others. Surprisingly, the World Development Reports rank only fifth. Top place goes to the research department’s Policy Research Report series. Second place goes to the Latin America and Caribbean Studies series.

Citation data suggest, then, that the Bank is a generator of knowledge—at least some Bank publications appear to have influenced scholars and authors of technical reports.

Does that mean that publications with low citation numbers had no impact? Not necessarily. In fact, they may well have had a larger impact on policymakers than the highly-cited books and articles. A book could reflect scholarly thinking rather than influencing it and yet still get read and still have an impact. To take an extreme example, the Bill and Melinda Gates Foundation was founded after Bill Gates was shocked by one the charts in the 1993 WDR on health; in a PBS interview, Gates said this was his “Aha” moment. 

By repackaging existing knowledge and enabling a complex subject to become accessible to policymakers and the public at large, many Bank books and reports have likely had an impact—quite possibly a very large one.

But how could it be measured systematically? I am trying out some ideas, but I’m also keen to hear from anyone who’s got good ideas. Anyone?

Oh, and the BBC story. It’s nice, but I’m a bit suspicious. I can find it only on the BBC’s Portuguese-language site (that’s odd), and only as an image (albeit with accompanying text). I also heard that the story might be a stunt. Or perhaps it was just that the artist was confusing the World Bank with someone else! Does anyone have one of those masks so we can check?
 

Comments

Submitted by Aniruddha Dasgupta on
Adam, great post. Some of us are worrying about exactly the same set of questions -- if knowledge is key for development, what kind of knowledge is more effective than others? Is it possible to measure Bank's performance in production and delivery of this knowledge? As this is a core service we provide as an organization, would it make sense to create some kind of a baseline so we could measure our performance systematically and compare across topics and years? This work is just starting; it is part of the work program of the knowledge strategy this year. it will be great to get you involved, let’s talk.

This is terrific. A few quick questions -- 1 -- How successful has the Bank's research been in reaching out to audiences beyond universities and policy makers? Is that audience worth engaging a bit more? Do the Bank's own operational teams make use of Bank research on a regular basis 2 -- Are there criteria other than citations (and the odd anecdote) that may be useful to measure the impact of Bank's publications? 3 -- Do you worry that about 30% of the Bank's publications are never cited at all? The number seems worse for book chapters in particular. Curious to see how the 70% number changes if you increase the number of citations threshold to 5, and if you discount citations by other Bank staff.