A World Bank Research milestone – Policy Research Working Paper No. 10,000

|

This page in:

“Public revenue is frequently earmarked by governments for … a specific … expenditure.  This paper examines whether [earmarking] appears to be effective in practice in achieving its objective of increasing expenditure.  Cross-section analysis shows that there is no statistically-significant difference in the proportion of road expenditure in gross investment between earmarking and non-earmarking countries.  Time series analysis lend[s] support that earmarking is associated with a somewhat higher degree of road expenditure.” 

Eklund, Per (1967).  “Earmarking of Taxes for Highways in Developing Countries” 
International Bank for Reconstruction and Development Economics Department Working Paper No. 1 

“This paper studies the impacts of the large-scale Road Sector Development Program in Ethiopia on local economic activity.  It exploits spatial and temporal variation in road upgrades across Ethiopia. The findings show that road upgrades contributed to increases in local economic activity.  However …  gains from road upgrades are concentrated in areas with moderate-to-high initial levels of economic activity. The results suggest that Ethiopia’s ambitious road infrastructure development program overall increased local economic activity and urbanization, but that it also had important distributional implications…”  

Alder, Simon, Kevin Croke, Alice Duhaut, Robert Marty and Ariana Vaisey (2022).  “The Impact of Ethiopia’s Road Investment Program on Economic Development and Land Use:  Evidence from Satellite Data”   
World Bank Policy Research Working Paper No. 10,000 

The two working papers1 quoted above serve as milestones along a path of over 50 years of economic policy research at the World Bank. The first quotation paraphrases from the introduction of the very first entry in the first official working paper series of the World Bank, the Economics Department Working Paper Series which began in 1967 and published 741 working papers through 1986. The second is from the abstract of the 10,000th entry in the World Bank’s Policy Research Working Paper (PRWP) series, which was published earlier this month.   

The PRWP series began in 1988 following the consolidation of several earlier specialized working paper series at the World Bank (see Figure 1). The PRWP series publishes research by World Bank staff and consultants with a stated mission to “disseminate the findings of work in progress and to encourage the exchange of ideas among Bank staff and others interested in development issues”. Over 8000 individuals have authored or co-authored PRWPs, a list that includes Nobel Laureates Douglass North, James Heckman, Elinor Ostrom, Joseph Stiglitz, Angus Deaton, Paul Romer, Michael Kremer, Esther Duflo, and Abhijit Banerjee  (click on their names to see their entries in the PRWP series).  The most prolific PRWP authors, in terms of most papers written, include Martin Ravallion, Asli Demirguc-Kunt, and David McKenzie, all current or former staff of the research department of the World Bank, with over 100 PRWPs each. 

Figure 1

Although separated by five-and-a-half decades of evolution in economic thinking, empirical techniques, data availability, and computational tools, the two papers excerpted above share features that are hallmarks of World Bank research. They address important practical questions facing development policymakers: how to finance road expenditures, and how roads affect local economic development. They both marshal empirical evidence based on modern techniques: both papers use difference-in-difference estimators (although the terminology was probably not in use at the time of writing of the earlier paper), among other empirical tools. The conclusions are nuanced: in the former, earmarking seems to matter within countries over time, but not across countries, while in the latter the effects of roads depend on initial conditions. The findings are caveated where needed:  the earmarking study notes the importance of influential observations in generating significant results, contains a discussion of causal interpretation, and has an annex discussion of spurious correlation, while the Ethiopia roads paper features a careful discussion of the limitations of each identification strategy.  Both papers take data access and replicability seriously: the full dataset is reproduced in an annex table in the earmarking paper, while the replication data and code for the Ethiopia roads paper are available on a World Bank GitHub repository here. Finally, both papers are connected to World Bank operations: the earmarking paper notes how the Bank’s lending in the sector facilitated access to the data used in the study, while the Ethiopia Road Sector Development Program studied in the second paper was supported by no fewer than five World Bank projects approved in 1998, 2003, 2004, 2007, and 2009.  

Of course, there are also important differences between these two papers. The earlier one features hand-drawn graphs on graph paper with typewritten labels, and given the costs of implementing the calculations for a single regression equation at the time, it seems unlikely that many specifications were estimated beyond those that appear in the paper.  The later one benefits from the full suite of modern tools and data, including detailed geospatial data based on satellite imagery (although it should be noted that even as early as 1971 the World Bank was beginning to explore the potential of satellite data for development policy analysis).  

A focus on development issues and developing countries 

While the PRWP series is younger and smaller than the NBER Working Paper Series (which began in 1973 and recently crossed the 30,000 paper mark) and the CEPR Discussion Paper Series (which began in 1984 and recently crossed the 17,000 paper mark), it is the largest working paper series focused on development economics and developing countries.  The next closest working paper series in terms of development focus is probably the IMF Working Paper Series with nearly 7500 entries since 1986.  However, given the IMF’s narrower topical focus (primarily monetary, fiscal, financial and external sector policy issues) and broader geographical mandate (including advanced economies), it does not have the same concentration on development economics and developing economies as the PRWP series. 

The PRWP series’ emphasis on development issues and developing countries is readily-apparent from Figure 2 and Figure 3, which track the evolution over time of the most frequent topics and countries covered in the PRWPs since 19882. Topics such as trade, finance, macroeconomics, and fiscal policy featured prominently in the PRWP series in 1988, together with education, health, and agriculture. The concentration of PRWPs across topics was fairly high:  trade alone accounted for 15% of the content of PRWPs in 1988.  Topics like poverty, the environment, social protection and governance were relatively less frequent in the early years of the PRWP series, but expanded in importance over time.  As of now, trade accounts for around 8% of the cumulative content of the PRWP series, while poverty represents 5% of the cumulative content.  Other topics such as governance, gender, and environment have also seen big increases in their coverage in the PRWP series over the past three decades.  Another notable pattern is the sharp increase in content related to data and surveys, which mirrors a trend in the wider economics literature to more empirical research described in Angrist et. al. (2017)

Figure 2

Figure 3

In terms of country coverage, in 1988 just four of the top 20 most-mentioned countries in the PRWPs were advanced economies (Japan at number 3, Germany at 15, France at 16 and the United States at 18).  Aside from Japan, the top five most-mentioned countries were large developing countries:  Mexico, Brazil, Thailand, and Indonesia. By 2022, China and India had joined Mexico, Brazil, and Indonesia to make up the top five most-mentioned countries in the PRWPs, while Japan (at position 13) was the only advanced economy remaining in the top-20 list.   Among developing countries there are disparities in country coverage:  East Asian, South Asian, and Latin American countries feature prominently in the top-20 list which also includes 6 economies in Sub-Saharan Africa – however no countries from the Middle East and North Africa region make the top-20 list. For a more systematic analysis of the frequency of economic research on developing countries, and the emphasis on advanced economies in top journals in economics, see this blog and paper (first published in the PRWP series of course…).  

Measuring PRWP influence (imperfectly) with downloads 

A 2014 PRWP posed the question “Which World Bank Reports Are Widely Read?” in its title, and answered the question with an analysis of download statistics available at the time for all World Bank reports (not just PRWPs).  Inevitably, the paper went on to become one of the most-downloaded PRWPs of all time.   

Downloads are a widely-used measure of the influence of research papers, because download data is readily accessible, and provides a rough indication of whether users are interested enough in a paper to click on it.  However, downloads are an imperfect measure, as some papers can be highly influential in a specific field or in a specific country without attracting widespread popular interest and associated downloads, while conversely communications effort devoted to a report can lead to high downloads but not necessarily high influence. 

With these caveats in mind, the full list of top-20 most-downloaded PRWPs appears in Table 1.  The left side lists top papers by total lifetime downloads.  Since older papers have had more time to accumulate downloads, the right side of Table 1 adjusts for this by listing top papers by downloads per year since publication3.

At first glance, these highly-downloaded papers are an eclectic mix, but a closer look reveals some patterns.  Papers that introduce new datasets are disproportionately represented in Table 1, accounting for 7 of the top 20 papers by total downloads, and 4 of the top 20 papers by downloads per year.  This is not so surprising:  users of popular datasets such as the Global Findex will naturally want to refer to the documentation of the data in the accompanying working paper for details, and users of the World Bank’s ubiquitous classification of low/middle/high-income countries will be interested in how these categories are defined

Papers that deal with highly-topical current events are also frequently represented in Table 1. This includes not just seven recent papers dealing with various aspects of the COVID-19 pandemic in the most-downloaded-per-year category, but a 2008 paper on the spike in global food prices that occurred in the same year. Papers that deal with issues of long-standing interest to the World Bank also appear among the most-downloaded papers. Most notably, five of the papers in the table deal with global poverty and global inequality and thus are tightly linked to the World Bank’s twin goals, while the PRWP with the most total downloads is a survey of the evidence on education and development.  And finally, for various reasons some PRWPs have attracted considerable media interest at the time of their publication, which translated into higher downloads as media consumers followed the links to the underlying papers. Examples of these include the 2014 paper on which World Bank reports are widely read mentioned above, as well as the 2020 paper on elite capture and foreign aid, which tops the list of most-downloaded papers per year of paper life with nearly 45,000 downloads per year on average over 2020 and 2021. 

Table 1

Top Papers by Total Downloads   Top Papers by Downloads Per Year
Total Downloads Year Published Paper   Total Downloads Per Year Year Published Paper
33,825 2008 Mitchell, Don (2008). "A Note on Rising Food Prices". PRWP No. 4682   5,691 2017 Demirguc-Kunt, Asli, Leora Klapper and Dorothe Singer (2017). "Financial Inclusion and Inclusive Growth: A Review of Recent Empirical Evidence". PRWP No. 8040.
35,457 2009 Kenny, Charles (2009). "Why Do People Die in Earthquakes? The Costs, Benefits and Institutions of Disaster Risk Reduction in Developing Countries." PRWP No. 4823    5,771 2019 Freund, Caroline; Alen Mulabdic; Michele Ruta (2019). "Is 3D Printing a Threat to Global Trade? The Trade Effects You Didn't Hear About". PRWP No. 9024
39,287 2014 Evans, David K.; Anna Popova (2014). "Cash Transfers and Temptation Goods: A Review of Global Evidence" PRWP No. 6886   5,859 2013 Lakner, Christoph; Branko Milanovic (2013). "Global Income Distribution: From the Fall of the Berlin Wall to the Great Recession". PRWP No. 6719
39,549 2001 Thomas, Vinod; Yan Wang; Xibo Fan (2001). "Measuring Education Inequality: Gini Coefficients of Education" PRWP No. 2525   6,239 2009 Kaufmann, Daniel; Aart Kraay; Massimo Mastruzzi (2009). "Governance Matters VIII: Aggregate and Individual Governance Indicators 1996-2008". PRWP No. 4978
40,288 2016 Fantom, Neil ; Umar Serajuddin (2016). "The World Bank's Classification of Countries by Income." PRWP No. 7528   6,694 2020 Espitia, Alvaro; Nadia Rocha; Michele Ruta (2020). "Covid-19 and Food Protectionism: The Impact of the Pandemic and Export Restrictions on World Food Markets". PRWP No. 9253
48,948 2008 Ravallion, Martin; Shaohua Chen; Prem Sangraula (2008). "Dollar a Day Revisited." PRWP No. 4620   6,715 2016 Fantom, Neil ; Umar Serajuddin (2016). "The World Bank's Classification of Countries by Income." PRWP No. 7528
49,867 2010 Beck, Thorsten; Asli Demirgyc-Kunt; Ouarda Merrouche (2010). "Islamic vs. Conventional Banking: Business Model, Efficiency and Stability." PRWP No. 5446   7,356 2018 Psacharopoulos, George; Harry Antony Patrinos (2018). "Returns to Investment in Education: A Decennial Review of the Global Literature". PRWP No. 8402
49,911 2009 Mu, Ren; Dominique van de Walle (2009). "Left Behind to Farm? Women's Labor Re-Allocation in Rural China". PRWP No. 5107   7,371 2021 Evenett, Simon J.; Bernard Hoekman; Nadia Rocha; Michele Ruta (2021). "The Covid-19 Vaccine Production Club: Will Value Chains Temper Nationalism?" PRWP No. 9565
52,730 2013 Lakner, Christoph; Branko Milanovic (2013). "Global Income Distribution: From the Fall of the Berlin Wall to the Great Recession". PRWP No. 6719   7,567 2021 Agarwal, Ruchir; Tristan Reed (2021). "How to End the COVID-19 Pandemic by March 2022". PRWP No. 9632
55,229 2012 Demirguc-Kunt, Asli; Leora Klapper (2012). "Measuring Financial Inclusion: The Global Financial Inclusion Indicators". PRWP No. 6025   7,661 2012 Milanovic, Branko (2012). "Global Income Inequality by the Numbers: in History and Now--An Overview--. PRWP No. 6259
57,713 2009 Beck Thorsten; Asli Demirguc-Kunt (2009). "Financial Institutions and Markets across Countries and over Time--Data and Analysis". PRWP No. 4943   7,689 2010 Schneider, Friedrich; Andreas Buehn; Claudio E. Montenegro (2010). "Shadow Economies All over the World: New Estimates for 162 Countries from 1999 to 2007". PRWP No. 5356
61,763 2014 Doemeland, Doerte; James Trevino (2014). "Which World Bank Reports Are Widely Read?". PRWP 6851   7,720 2014 Doemeland Doerte; James Trevino (2014). "Which World Bank Reports Are Widely Read?". PRWP 6851
63,311 2009 Bhattacharyya, Subhes C.; Govinda R. Timilsina (2009). "Energy Demand Models for Policy Formulation: A Comparative Study of Energy Demand Models". PRWP No. 4866   8,663 2010  Kaufmann, Daniel; Aart Kraay; Massimo Mastruzzi (2010). "The Worldwide Governance Indicators: Methodology and Analytical Issues". PRWP No. 5430
76,435 2020 Maliszewska, Maryla; Aaditya Mattoo; Dominique van der Mensbrugghe (2020). "The Potential Impact of COVID-19 on GDP and Trade: A Preliminary Assessment". PRWP No. 9211   8,712 2020 Schellekens, Philip; Diego Sourrouille (2020). "COVID-19 Mortality in Rich and Poor Countries: A Tale of Two Pandemics?". PRWP No. 9260
76,612 2012 Milanovic, Branko (2012). "Global Income Inequality by the Numbers: in History and Now--An Overview--. PRWP No. 6259   9,122 2020 Apedo-Amah, Marie Christine; Besart Avdiu; Xavier Cirera; Marcio Cruz; Elwyn Davies; Arti Grover; Leonardo Iacovone; Umut Kilinc; Denis Medvedev; Franklin Okechukwu Maduko; Stavros Poupakis; Jesica Torres; Trang Thu Tran (2020). "Unmasking the Impact of COVID-19 on Businesses: Firm Level Evidence from Across the World". PRWP No. 9434
81,106 2009 Kaufmann, Daniel; Aart Kraay; Massimo Mastruzzi (2009). "Governance Matters VIII: Aggregate and Individual Governance Indicators 1996-2008". PRWP No. 4978   10,183 2007 Hanushek, Eric A.; Ludger Woessmann (2007). "The Role of Education Quality for Economic Growth". PRWP No. 4122
89,583 2020 Andersen, Jorgen Juel; Niels Johannesen; Bob Rijkers (2020). "Elite Capture of Foreign Aid: Evidence from Offshore Bank Accounts". PRWP No. 9150   11,520 2021 Aklin, Michael; Andreas Kern; Mario Negre (2021). "Does Central Bank Independence Increase Inequality?" PRWP No. 9522 
92,265 2010 Schneider, Friedrich; Andreas Buehn; Claudio E. Montenegro (2010). "Shadow Economies All over the World: New Estimates for 162 Countries from 1999 to 2007". PRWP No. 5356   11,617 2020 Demirguc-Kunt, Asli; Alvaro Pedraza; Claudia Ruiz-Ortega. (2020). "Banking Sector Performance During the COVID-19 Crisis". PRWP No. 9363
103,958 2010  Kaufmann, Daniel; Aart Kraay; Massimo Mastruzzi (2010). "The Worldwide Governance Indicators: Methodology and Analytical Issues". PRWP No. 5430   38,218 2020 Maliszewska, Maryla; Aaditya Mattoo; Dominique van der Mensbrugghe (2020). "The Potential Impact of COVID-19 on GDP and Trade: A Preliminary Assessment". PRWP No. 9211
152,749 2007 Hanushek, Eric A.; Ludger Woessmann (2007). "The Role of Education Quality for Economic Growth". PRWP No. 4122   44,792 2020 Andersen, Jorgen Juel; Niels Johannesen; Bob Rijkers (2020). "Elite Capture of Foreign Aid: Evidence from Offshore Bank Accounts". PRWP No. 9150

 

How does the distribution of downloads look beyond the “tail” of such highly-downloaded papers?  Figure 4 summarizes the data and also provides a comparison with the NBER Working Paper Series, which is the largest and highly prestigious working paper series in economics. The figure shows the distribution of downloads per year across all PRWPs (left panel)4 and NBER Working Papers (right panel).4  Starting at the top end of the distribution, around 4 percent of PRWPs have more than 1000 downloads per year. The much larger NBER working paper series has a higher share of “greatest hits”, with 8 percent of papers gathering more than 1000 downloads per year. There are also some differences at the bottom end of the distribution: about 10 percent of PRWPs have 20 or fewer downloads per year, while the corresponding figure for the NBER is 2 percent. Aside from these differences at the top and at the bottom, the distribution of the roughly 90 percent of remaining working papers is quite similar across the two series. This suggests a comparable pattern of influence, subject of course to the caveats around this measure noted above. 

Figure 4:  Working Paper Downloads 

Working Paper Downloads

Note: The height of each bar shows the share of working papers with the range of downloads per year indicated on the horizontal axis.  Each bar is an increment of 20 downloads.  The last bar shows the fraction of papers with more than 1000 downloads per year.  Population refers to papers published between 1988 and 2021.

The next 10,000 working papers 

It took 34 years since 1988 for the PRWP series to publish its first 10,000 working papers. After a slightly slower pace at the beginning, since 2000 the PRWP has published around 350 papers year.  At this rate, PRWP No. 20,000 will be published somewhere around 2050.   

Much is likely to change in development economics research over the next three decades, just as much has changed since the days of hand-drawn graphs and typewritten manuscripts in  IBRD Economics Department Working Paper No. 1 back in 1967.  Much is also likely to change in the technology available to empirical researchers in development economics, when one considers that in 1988 when the PRWP series began, the Intel 386 processor was the industry standard for personal computers, laptops were an exotic item, and backup storage was in the form of 3.5in floppy disks with 1.4MB of capacity apiece.  The availability of data and the conceptual tools to analyze it are also likely to continue to change drastically, when one considers that empirical analysis in IBRD Economics Department Working Paper No. 1 was based on ordinary least squares performed on a grand total of 47 country-year datapoints, while today it is not uncommon for datasets to run into the millions of observations and sophisticated machine learning algorithms are routinely deployed alongside more traditional econometric techniques. 

However, it is likely that many aspects of PRWP No. 20,000 will be quite familiar to us today, just as many features of IBRD Economics Department Working Paper No. 1 resonate today more than 50 years after it was written.  These durable features of applied development economics research at the World Bank will continue to be a focus on practical questions relevant to policymakers, carefully nuanced and caveated empirical analysis, a premium on identifying causal effects, and a close connection to World Bank operations. 


1. Roula Yazigi provided the same outstanding logistical support in preparing this blog as she has provided to the PRWP series since 2007.  Thanks also to Luiza Cardoso, Alice Duhaut, Olivier Dupriez, Elisa Liberatori-Prati, Norman Loayza, Bob Malloy, and Carmen Reinhart for helpful comments. 

2.  I am very grateful to Olivier Dupriez, Aivin Solatorio, and Kamwoo Lee for the text analytics they performed to generate these graphs.  The country names frequency is extracted from full-text searches of all PRWPs, and the graph reports the count of mentions of each country across all PRWPs.  The topics analysis is based on unsupervised topic analysis of the full text of each PRWP, and then the top topics were then manually labelled (and in some cases combined) for clarity as shown in the graph.  The data reflects the proportion of the content of each PRWP assigned to each topic by the unsupervised analysis.  The total proportions sum to less than one since the top 20 topics account for only 60 percent of the total content. 

3. The downloads data described here aggregate downloads from three distinct platforms: The World Bank’s “Documents and Reports (DR)” and “Open Knowledge Repository (OKR)”, as well as the “Social Science Research Network (SSRN)”.  I am very grateful to Luiza Cardoso for scraping the download data from SSRN, which captures downloads from that platform since the mid-2000s.  The download data from the OKR (DR) include downloads since 2012 (2014), and the OKR contains only documents published from 2002 onwards. Downloads per year of publication exclude papers published in 2022.  The download data is current as of late January 2022. 

4. A couple of notes on the data are important.  NBER download data is available since 2011, and covers all NBER papers back to the 1970s.  To make this more comparable with the PRWP data, (a) I consider only downloads since 2014 (since the Documents and Reports downloads data starts in 2014, and (b) I drop NBER papers before 1988 since this is the year the PRWP series starts.  I am very grateful to Dan Feenberg at the NBER for sharing the NBER download statistics.  

Authors

Aart Kraay

World Bank Group Deputy Chief Economist and Director of Development Policy, Development Economics

Join the Conversation