Syndicate content

Whither the development agency’s flagship report?

Adam Wagstaff's picture

The Economist carried a couple of stories recently about how two hitherto major institutions in my home country (newspapers and pubs) have been forced to adapt in the face of changes in public preferences. Many didn’t—as a result newspaper circulation and pub numbers have both fallen dramatically. The newspapers and pubs that did survive operate very different business models from the newspapers and pubs in existence even 10 years ago.

Some data I’ve assembled make me wonder whether—like the newspaper and pub—the development-agency flagship might not also be an institution in need of reform.

The flagship

Most big development and international agencies have a flagship. The World Bank launched its World Development Report in 1978. The IMF’s World Economic Outlook started two years later. The UNDP launched its Human Development Report in 1990, and WHO followed with its World Health Report five years later. Several other UN agencies have annual or periodic flagship reports too.

The production of all the reports involves teams assembled for the purpose, comprising a mix of senior and junior staff and consultants. Most seem to take around a year to produce and disseminate, and are presumably therefore quite costly. Most focus on a specific theme, but also take the opportunity to present a roundup of new data though the data are usually available elsewhere too and are not always produced specifically for the report. All have a broad audience in mind, and aim at creating new knowledge in addition to synthesizing existing knowledge. All seek to influence the way policy makers, the public and scholars think about the chosen topic.

Given the costliness of these reports, it makes good sense to ask whether they are indeed influencing people, and whether if they have done so in the past they continue to do so today. This isn’t easy, but the huge volume of data on the internet gives us some clues about influence—and about interest which seems likely to be a precondition to influence.

Gauging interest

Fig 1 shows a 20-week moving average of the number of internet searches for four flagship reports that I assembled using Google Trends. The data are normalized with the first-entered search term averaging 1 over the seven-year period, and the other searches adjusted accordingly. On average the search frequency for the WDR was 60% that of the HDR, while the frequency of searches for the WHR and WEO were 31% and 21% of the HDR. Searches for the HDR have dropped considerably since 2004. Searches for the WDR have declined as well though less spectacularly. By contrast searches for the WEO increased during 2007 and stayed at the same level thereafter. Searches for the WHR started falling at the start of 2006 after an initial rise.

The internet search data suggest there’s been something of a waning interest in at least three of the flagships. This may reflect what seems to be a waning interest in international development. Fig 2 shows that searches for “international development” have fallen throughout the period, as have searches for related terms. As Fig 3 shows, German cars haven’t suffered the same fate: in fact, interest in the Mini—which has been made by BMW since 2001—has grown since the new model was launched in 2007. And searches for new technologies have risen pretty much continuously during the period (Fig 4). So if people had been growing ever more interested in international development and flagship reports, I think the data would have picked it up.

 

 

Gauging influence

Waning interest doesn’t necessarily imply waning influence. But how to measure influence? Martin Ravallion and I have argued for assessing influence on development thinking using citations in documents captured by Google Scholar. Unlike the Web of Science, Google Scholar captures not just established journals but also new journals, online journals, books, dissertations, and technical reports. Fig 5 shows for each year the number of hits in Google Scholar for each of the flagships. The year refers not to the year the flagship was published but the year of the article or book citing it. The charts don’t distinguish between flagships from different years. So it’s not surprising that in 1990 the WDR was getting more citations than the HDR, because by 1990—the first year of the HDR—12 WDRs had already been published. And it’s not surprising if citations keep going up. A fairer comparison across flagships and over time is to adjust citations in any year by the number of flagships to date to get an average figure per report. This chart (Fig 6) suggests that the influence per report continued to grow for all four flagships through to around 2003 but has since fallen off except in the case of the WEO.

 

Getting at influence more broadly—among policy makers and the public at large—is harder. Coverage in the media is one way. Fig 7 shows trends in Google News items for the four flagships. Around 2000 the WDR and WHR got left behind by the HDR and WEO. Then the “great recession” put the WEO on a trajectory reminiscent of Facebook’s trajectory in Fig 4. Of course news items reflect not just interest and influence but also the efforts of communications officers of the agencies. The UNDP might have just gotten much more media-savvy very quickly. And of course in any case Google News data don’t get behind the scenes and tell us whether people in power had their thinking fundamentally altered by a flagship report. We know the 1993 WDR shocked Bill Gates into setting up the Bill and Melinda Gates Foundation. But there’s no blip in the Google News data reflecting this.

The last five years—and the future

I like the Gates WDR story. But at heart I’m a numbers person. I find it striking that when you look at Figs 1 and 2 and the comparable period of Fig 6 you see a waning of interest in flagship reports and of their influence among academics and report-writers. When British newspaper- and pub-owners saw waning interest in their wares, they did a radical rethink. Is it time for development agencies to do a rethink on their flagships? Is the flagship model outdated? Could a different model help reverse the apparent decline in interest in international development?

Comments

Adam: Thanks for opening up this discussion, and for the very interesting findings on WDR (and other flagships') readership and interest. It could be that in the modern era of blogs, Facebook and Twitter, not to mention quick-turnaround electronic journals and working papers, the notion of a glossy, well-written flagship report is losing its appeal. Perhaps people don't get their knowledge from carefully-written and well-researched books (one-way interaction), but rather through engaging with other scholars and practitioners as they grapple with specific problems (two-way interaction). If this is true, then we should be thinking of flagships not as a finished product alone, but as a process. Of course, we now undertake consultations in preparing WDRs, but the final product is still a one-way interaction. Why not consider a flagship that is a process by which scholars and practitioners from around the world engage with Bank staff on pressing development issues? As I have mentioned on my blog (https://blogs.worldbank.org/africacan/development-30-0), we could produce the WDR like wikipedia, with everybody able to contribute and correct other people's contributions. Then it will be the "World's Development Report." Regards, Shanta

Shanta, thanks. I think what you say about a shift from one-way to two-way interaction applies in a lot of spheres. I suspect newspaper circulation dropped partly because people want to get their news and views from multiple sources, and partly because they want to debate ideas not read passively what “experts” think. I agree with you that this trend has happened too in international development. The idea of a wiki-WDR—the World’s Development Report—is really innovative. Would you adopt Wikipedia rules? Or would you handpick “authors”? Or could interested authors apply? Would you limit the number of authors? Would they get paid? Who would arbitrate in the event of disagreements? Would there be a lead “author” with the final say? A World Bank staffer? All the best, Adam

Submitted by Syed Akhtar Mahmood on
Shanta: I endorse your view about looking at flagship reports as part of a process. I think this would be true of flagship reports at the country level as well. Like the WDR, we increasingly consult with people during the production of the report but the process ends with the publication and one-round (often one-event) dissemination of the report. The "wiki WDR" idea can be applied to selected country reports as well. Regards, Akhtar

Adam, Akhtar: Thanks for the encouragement. I haven't thought through the mechanics of a wiki-WDR (maybe we can work them out on this blog), but here are some thoughts. Adopting wikipedia rules, the "core team" of the WDR (who could be from inside and outside the Bank) would define the topics of the report, and then invite contributions, as in wikipedia. Ideally, these topics would be ones where there is some, genuine disagreements (such as industrial policy, just to pick an example). Anybody could contribute, but we may handpick one or two leading figures to get the discussion going. The important point is that it's not a discussion (where multiple people contribute their views, as in this blog for instance), but rather a report, where people continuously edit and re-edit each others contributions. Some scientific journals now do this. If the discussion doesn't "converge" (this is the term the physicists use), then we would agree to have two write-ups, each presenting a side of the debate. Each of these would then be edited and re-edited by anybody who so desires, although the core team would reserve the right to reject particular edits (as they do in wikipedia). While this works for physics journals where the audience and readership is limited, we would like to get a broader group of participants in a wiki-WDR, including if possible, poor people themselves (the people whom we're writing about in other WDRs). I'm not sure how to do it, but one idea is to make use of the fact that a majority of the poor now have access to cell phones. If there is some way we can get their voices incorporated in the WDR through text messaging, then it would truly be the World's Development Report. Shanta

Adam: interesting. If you run the four reports through Google's Books Ngram Viewer you get a slightly different story: http://bit.ly/ho4rbH The data there are still very beta, and only goes up to 2000, but the WDR seems to have hit its peak in 1994, but is still much more widely cited than the other three. Newspapers started to lose readers for a number of reasons, not least of which was that the format no longer reflected their changing lifestyle: bulky, big, physical, late, hard to share, not easy to digest in an increasingly crowded and intrusive world. I'd suggest that the lesson that most applies to the publications you study here is not so much that people seek differing views but that they need those views pithy and fast. Stories have gotten shorter, sharper, clearer, more succinct. (An interesting study would be the length of sentences over the years.) A big glossy report is anachronistic because that's not the best way to share information. A Wikipedia approach, though interesting to experiment with internally, doesn't address these issues.

Jeremy: thanks. We think alike. I wonder why! I had also done the Ngram for these reports. I excluded it because unlike Google Scholar’s, the Ngram sample excludes journals, technical reports, etc., and because according to Michel et al.’s Science article(http://bit.ly/hwziQC), the Ngram sample includes “only” 5 million of the 15 million books that Google has digitized (15 million is around 12% of all books ever published). As you say, the picture that emerges is a little different: it shows the WDR in a much better light, but an earlier peak of influence.

Akhtar, Shanta, Jeremy: thanks for this discussion. Several ideas are emerging here. (1) Process matters. The model emerging is one of discussion—and an open discussion at that. (2) Having an outcome matters. In a blog, people share viewpoints, but then get on with their lives. By contrast, the open discussion needs to lead somewhere: it might converge in which case we’d end up with a shared viewpoint; or it might not converge in which case we should present the diverging viewpoints. (3) Whether we’re talking about the process or the outcome, we need to avoid being bulky and physical, late, hard to share, and difficult to digest. The views need to come “pithy and fast”; they need to be short, sharp, clear, and succinct. This is all nicely in synch with the Bank’s Open Development philosophy. As Bob Zoellick (http://bit.ly/aNL5UQ) put it “…development economics… must reach out to better encompass the experiences of successful emerging economies not with ordered templates or with blueprints, not with prescriptions for prescriptees, but inquiringly, cooperatively, openly.” It’s actually not very far from the vision for the WDR that Joe Stiglitz (http://bit.ly/hqxTri) had when he was Chief Economist: “…the objective of the WDR was to begin a global dialogue, a democratic conversation about some of the most contentious issues in development. It did not bother me that we might not know the right answer. Indeed, it bothered me more that we sometimes pretended to know more than we did. To me, the role of an outside adviser was more to ask the right questions—or to help those in the developing countries ask those questions—than to give the right answer.” Shanta’s wiki approach fits (1) and (2) nicely. It permits an open discussion and pushes toward an outcome. But As Jeremy notes, the wiki approach only partly fits (3). It has the merit of being easy to share. But the outcome would likely be longer, harder to digest, less readable, etc. than the existing WDR, which while not faultless usually reads rather well, especially in comparison to a typical Wikipedia entry which often reads like it was written by a group of 9th graders. I wonder whether the new open-access open-assessment e-journal Economics (http://www.economics-ejournal.org/) might not be a model. The process would start with an open online discussion along the lines Shanta suggests. At the close of this, an editor would summarize the discussion in accessible prose, and would solicit discussion papers on specific topics. These discussion papers would be just that—papers that get posted online, with space for people to comment on them. They would also be sent to “reviewers” whose reports would also be posted online. The editor would summarize in plain English each of these papers, and the online and reviewer comments. The editor would invite revisions from some of the authors, and might also solicit new papers, e.g. where it was clear gaps had been left. Where a consensus has emerged between authors, the editor might solicit amalgamations of papers. Where dissention exists, he or she would solicit dissenting final papers. These final papers would become the source material for a package of multimedia materials: short accessible online pieces, virtual and in-person debates, videos, podcasts, etc. We could call this new model the “World Development Debate”.

Dear Adam, Those of us who work on the Human Development Report (HDR) welcomed this blog, which graciously acknowledges the relative strength of the HDR vis-à-vis the World Development Report in recent years, as measured by the frequency of Google searches for the respective titles. But we would take issue with your overall hypothesis. You contend that regardless of which report is more frequently sought out, overall “interest” in such reports is steadily declining, as indicated by the contrast between the HDR and WDR search profiles since 2005, illustrated through Google Tools. Your conclusions misconstrue what those Google numbers actually show, which is the relative use of these terms in google searches over time in the ever-expanding universe of web-searches. As Google notes, “All numbers are relative to total traffic.” These traffic patterns in turn reflect the exponential growth of (among other things) both users and providers of mainstream pop-culture content on the web. Likewise it is not surprising that there is relatively more Google-searching today for Eminem, say, than for a Joe Stiglitz, than there was five years ago. But that doesn’t mean that Joe’s audience has been shrinking in absolute terms. We believe that this flawed analysis does a disservice to both publications. It certainly does to ours. We know this empirically, as traffic to the HDR website continue to expand, with downloads of the Report and page views of the site both almost doubling over the past two years (there were 180,000 downloads of the 20101 HDR in the two months after the November 4th launch, more than triple the size of our print run). This growth is consistent with the steady, significant expansion in global news coverage of the HDR in recent years, in all major languages. If interest is measured by actual reading and referencing of the Report, interest has never been higher. Moreover, the Google-search metric doesn’t capture the rising proportion of available links to the HDR (and the WDR and similar resources) that don’t require the intermediary of a search engine. News stories on the web now routinely include embedded hyperlinks. And many browsers recognize and route directly to common generic names through the URL window. Bookmarking on browsers is the habit of frequent site visitors. Links are increasingly commonly stored and shared through social networking sites like Facebook, which recently surpassed Google as the most widely used U.S. website. Out own internal web data shows the majority of visitors to our website arrive through such ‘direct’ links, rather than the redirected access triangulated through a Google (or Yahoo or Bing) search and are hence not capture by the Google numbers. I think that we converge in the belief that we all need to be continually thinking about the role of different products and services that the World Bank and other development agencies produce. It is obviously a fast changing world, and we need to ensure that what we offer is interesting, relevant and as accessible as possible to our audiences around the world. But it would be premature to consign to history the concept and goals upon which the concept and goals upon which reports like the HDR and WDR are based. All the best Jeni

Dear Jeni, Thanks for the comment. I was also struck by the relative strength of the HDR. You’ve worked in both institutions so you probably have some hypotheses to explain the data. My hunch is that the HDR’s heavy engagement at country level helps the HDR get traction. As for my alleged misinterpretation of the Google Trends data, I concede that Google isn’t terribly clear in its help file (http://bit.ly/1mHmV) on exactly what the data show. So, there’s a nonzero probability you’re right! But I think there’s a higher probability I’m right. Let me explain what I think Google Trends does, and then why I think my explanation is correct. When you enter a search term (say, UNDP) the chart’s legend shows a 1.0 against the search term UNDP (you need to sign in), and at the top of the chart you see “Scale is based on the average worldwide traffic of undp in all years.” If you export the data (with relative scaling), you’ll find they average to 1. I think they’ve taken the number of searches for UNDP in each time period and divided it by the average searches for UNDP over the time period (i.e. total searches for UNDP divided by the number of periods). This averages to 1. Suppose you now add a second term (say World Bank). At the top of the new chart you still see “Scale is based on the average worldwide traffic of undp in all years” and against UNDP you still see 1. Against World Bank you’ll see 3.25, which the help file tells us means that people have searched for “World Bank” 3.25 times as often as they have searched for “UNDP” over the period in question. So, what they’ve done here is to divide the number of searches for “World Bank” in each period by the average searches per period for UNDP in the time period in question. The average number of searches for the first search term serves as the numeraire for all search terms. Each non-numeraire series averages to the total number of searches for the term (e.g. “World Bank”) as a proportion or multiple of the total number of searches for the numeraire search term. The help file is almost 100% consistent with my interpretation: “The data is scaled based on the average search traffic of the term you’ve entered… For example, if you entered the term dogs, the graph you’d see would be scaled to the average of all search traffic for dogs from January 2004 to present.” “The data is scaled based on the average search traffic of the term you’ve entered.”“All results from Google Trends are normalized, which means that we’ve divided the sets of data by a common variable.”“The numbers you see on the y-axis of the Search Volume Index (which you can see after you’ve signed in to your Google Account) aren’t absolute search traffic numbers. Instead, Trends scales the first term you’ve entered so that its average search traffic in the chosen time period is 1.0; subsequent terms are then scaled relative to the first term.” Where the help file is confusing is where it says: “Google Trends analyzes a portion of Google web searches to compute how many searches have been done for the terms you enter, relative to the total number of searches done on Google over time”, and “Note that all numbers are relative to total traffic.” But I think what they mean here is not total searches, or total traffic for all possible search terms, but rather total searches for your numeraire search term. There are two reasons why I think I’m right. First, if you’re right, Google Trends doesn’t actually tell us anything about trends! Google’s a pretty smart company, so why would they put out a product that doesn’t do what its name makes you think it does? Of course they probably don’t want to give you actual numbers for business reasons. But they don’t need to normalize by the universe of searches to conceal actual numbers. They can do that by doing what I’m claiming they have done. We can’t back out from these numbers the actual number of searches; using the first search term as the numeraire allows them to keep secret the actual number of searches. Second, suppose they did—as you claim—divide through each period’s search number for each term by total Google searches over the period in question. Then the data for the first search term wouldn’t average to 1: it would average to 1 divided by the average number of all searches, which would be a tiny number. And the data for the second term wouldn’t average out to the total number of searches for the second search term as a fraction or multiple of the total number of searches for the first search term, which is what Google tells us is what it is. So, my sense is that the numbers do indeed show what I claim they show: a downward trend in the number of Google searches for our flagship reports. Incidentally, searches for “Stiglitz” haven’t actually trended downwards. “Stiglitz” has dominated “World Development Report” (http://bit.ly/edPXON) each year since the start of the Google Trends data. In 2005 the gap wasn’t that big. “Stiglitz” kept steady, and by 2010 was being searched for many times more than “World Development Report”. And by the way, searches for “Eminem” haven’t really trended upwards. “Bono” has also stayed pretty flat. By contrast, interest in “Jeffrey Sachs” seems to be waning. I also think it’s a little dangerous to gauge interest via news stories. As I said in the original post, news items reflect not just interest and influence but also the efforts of communications officers of the agencies. What we’d like are data showing numbers of people actually reading articles about the reports. Where I concede you’re onto something is people bookmarking urls, referrals from Facebook, intelligent browsers, etc. So, people may well be relying less on Google than they did in the past. But shouldn’t this also be true of “Stiglitz”, “Bono”, and “iPhone”? And even more true of “Facebook”? Yet all these have managed a flat Google Trends chart, or a rising trend. And wouldn’t a curious reader want to know what people were saying about the report they’ve just downloaded? And wouldn’t we want to see people actively searching out multiple and new viewpoints, rather than lazily relying on a few favorite sources via RSS feeds and tweets? So, my view is still that a downward trend in a Google Trends chart is a warning sign of waning interest. Take care, Adam

Submitted by Keith Hansen on
Thanks to Adam for kicking off this thought-provoking exchange. I have little doubt that interest in long, formal works is declining, and I suspect the effect is not limited to development publications. Even had interest in development remained constant, the sheer increase in the number of "publications" (including everything on the Internet) over the past decade should presumably have had some crowding out effect on readership of any individual item. And while definitive proof remains wanting, there is plenty of circumstantial evidence that attention spans are steadily diminishing. People increasingly want to acquire their information a la carte, not from a prix fixe menu of someone's else's choosing. That said, although it is a foolish person indeed who tangles with Adam on research methodology, I have a question. Simply eyeballing your Figs. 1 and 2 on my small laptop screen, the slopes of declines look broadly similar. So how do we know that the waning searches for WDRs are not simply a proportionate subset of the waning searches for development in general? And if they are, how do we know that flagship reports are losing their relative 'market share' compared to other development publications? Granted, there may be a causal relationship here: the heavy preference for thick flagship reports by development agencies may be driving people away from the subject in general. But I'm not sure we can conclude from these figures that flagships are losing their effectiveness relative to any other form of communicating about development. I also wonder whether some of this reflects the explosion of secondary sources and alternate routes to information. For example, someone who would once have searched for the WDR might now find it summarized, described, and hyperlinked on a development wonk's blog site, and thus never have to enter a search term to find what they need. Those nuances aside, however, I suspect your hypothesis is correct, and am persuaded that at least for lay audiences, some fresh thinking is due. This reminds me obliquely of an experiment a journalist did 20 years ago. He wanted to test whether people actually read the "prestige" books that become briefly popular each year and that every smart person wants to be seen as owning. My numbers may be off, but my memory is that he went to several bookstores in Washington and inserted a credible note with a phone number and an offer of $10 at page 138 in about 200 copies of "A Brief History of Time." Only a small handful of people ever called. I wonder if there's a digital equivalent we could try?

Wonderful story at the end, Keith! Trust you to make us laugh out loud and be thought-provoking at the same time! Actually, I deleted a sentence at the end that acknowledged the point you made—the rates of reduction for “World Development Report”, “International Development”, etc. are indeed rather similar. My deletion was the analogue of $10-bill offer slipped into the book! You not only read to the end; you also spotted the missing sentence! So, I think you qualify for at least a drink! And you’re right—the waning of interest in flagships may just be a reflection of (and certainly hasn’t necessarily caused) a dwindling of interest in international development. But as you and Shanta some to agree, it could well be that some fresh thinking on our flagships could help rekindle interest in development. But what to do? Perhaps we could slip into the next WDR at page 138 a small questionnaire soliciting ideas, promising a raffle and a large prize for the lucky returned questionnaire?