A couple of weeks ago, I came across a fresh World Bank working paper
(Doemeland & Trevino 2014) that examined downloads and citations for World Bank policy reports. The paper reports that 31 percent of policy reports have never been downloaded and 87 percent have never been cited. I thought this was an interesting statistic and – since it was publically available – I opened up Twitter and
tweeted it. To the statistic, I added the question as to whether we needed “more relevant reports or better dissemination,” hoping to spark a conversation. At that moment, I had 40 followers on Twitter. In case you’re not familiar with Twitter, 40 followers is
not very many. The
median number of followers among people who have tweeted in the last 30 days is about 61.
Later that day, another Twitter user
sent out a “modified tweet”. He quoted part of my tweet (with appropriate sourcing) and added his own (less positive) commentary. This user, Justin Sandefur, has about 2,600 followers. The next day, Tyler Cowen, who co-authors the popular economics blog Marginal Revolution,
posted the statistics with a reference to Sandefur’s tweet. The post had 30 comments as of this morning, so we know people read it.
Christopher Ingraham read the post on Marginal Revolution and went on to post the statistics on the Washington Post’s Wonkblog with a more positive interpretation, highlighting that this is likely a problem for Washington think tanks broadly and commending the Bank for examining and publishing the numbers. He had 55 comments as of this morning; again, so we know people are reading it. Ingraham’s blog post was republished in the Sydney Morning Herald, excerpted on Slate, and referenced elsewhere. (A blog dedicated to “content access”, Xillio.com, even offered the Bank some tips on increasing access to our reports; Thanks, Xillio!)
Since all this has happened, I’ve had several conversations that highlight how these statistics may be misleading. Citations aren’t really the right measure of impact for policy reports, since government policy makers aren’t writing citation-heavy academic papers (thankfully!). Even downloads offer an underestimate: Many of these reports are emailed or hand-delivered directly to the policy makers most likely to use them (i.e., for a report on health in Malawi, to the Minister of Health in Malawi and her staff).
As Doemeland and Trevino write, “Many policy reports were … prepared to assess very specific technical questions or inform the design of lending operations.” The majority of these are not research working papers but rather reports like “ Pakistan - Finding the path to job-enhancing growth” and “ Education in Oman: The Drive for Quality.” Still, there is clearly room for improvement. But what I take away from this experience is less about the statistics themselves and more about how to get research results into public debate. (I admit, I’d have preferred to learn this with some of my own research results, say on how to reduce poverty in Tanzania, but maybe next time.)
This is a relatively uncommon situation, where we can clearly trace the growth in awareness from my minimally circulated tweet all the way to the Washington Post and beyond. We can’t rule out that the stat would have gone viral anyway; maybe Sandefur would have encountered it some other way. But we at least see clearly the distribution path that this piece of evidence followed.
From this, we see the importance of “connectors” in getting our knowledge work into the public dialogue. A connector, per Malcolm Gladwell’s book The Tipping Point, is a person who connects with lots of other people. Sandefur is a connector, with more than 2,600 followers, putting him above the 98th percentile of active Twitter users. Cowen is also a connector.
If you want to get your results known, get them into the hands of connectors. Oh, and it doesn’t hurt if the results are interesting. (As of early this week, Doemeland & Trevino had more than 850 downloads, which puts them on the far right tail of World Bank policy reports, per the figure from their paper.)
Bonus reading: The population of reports that Doemeland & Trevino draw on is available at the World Bank’s Documents & Reports archive, under “Economic & Sector Work.”
Christopher Ingraham read the post on Marginal Revolution and went on to post the statistics on the Washington Post’s Wonkblog with a more positive interpretation, highlighting that this is likely a problem for Washington think tanks broadly and commending the Bank for examining and publishing the numbers. He had 55 comments as of this morning; again, so we know people are reading it. Ingraham’s blog post was republished in the Sydney Morning Herald, excerpted on Slate, and referenced elsewhere. (A blog dedicated to “content access”, Xillio.com, even offered the Bank some tips on increasing access to our reports; Thanks, Xillio!)
Since all this has happened, I’ve had several conversations that highlight how these statistics may be misleading. Citations aren’t really the right measure of impact for policy reports, since government policy makers aren’t writing citation-heavy academic papers (thankfully!). Even downloads offer an underestimate: Many of these reports are emailed or hand-delivered directly to the policy makers most likely to use them (i.e., for a report on health in Malawi, to the Minister of Health in Malawi and her staff).
As Doemeland and Trevino write, “Many policy reports were … prepared to assess very specific technical questions or inform the design of lending operations.” The majority of these are not research working papers but rather reports like “ Pakistan - Finding the path to job-enhancing growth” and “ Education in Oman: The Drive for Quality.” Still, there is clearly room for improvement. But what I take away from this experience is less about the statistics themselves and more about how to get research results into public debate. (I admit, I’d have preferred to learn this with some of my own research results, say on how to reduce poverty in Tanzania, but maybe next time.)
This is a relatively uncommon situation, where we can clearly trace the growth in awareness from my minimally circulated tweet all the way to the Washington Post and beyond. We can’t rule out that the stat would have gone viral anyway; maybe Sandefur would have encountered it some other way. But we at least see clearly the distribution path that this piece of evidence followed.
From this, we see the importance of “connectors” in getting our knowledge work into the public dialogue. A connector, per Malcolm Gladwell’s book The Tipping Point, is a person who connects with lots of other people. Sandefur is a connector, with more than 2,600 followers, putting him above the 98th percentile of active Twitter users. Cowen is also a connector.
If you want to get your results known, get them into the hands of connectors. Oh, and it doesn’t hurt if the results are interesting. (As of early this week, Doemeland & Trevino had more than 850 downloads, which puts them on the far right tail of World Bank policy reports, per the figure from their paper.)
Bonus reading: The population of reports that Doemeland & Trevino draw on is available at the World Bank’s Documents & Reports archive, under “Economic & Sector Work.”
Join the Conversation