How can we measure state capacity? Do you start upstream or downstream?

About a year ago, Frank Fukuyama released an article entitled “What is governance?” in the Governance journal that became an “instant classic” in the field. Within a month it had elicited over 15 responses from prominent scholars on the Governance blog, not to mention commentary posted elsewhere—including this blog. It already has over 40 google citations, including articles in Spanish, Italian and Portuguese. And a month ago, Governance journal published two more commentaries on Fukuyama’s original article (by Robert Rotberg and Craig Boardman), reinvigorating the debate.

Basically, the “world” (well public management practitioners and academics, at least) seems to be dividing into two camps. The first group is those who think that state capacity should be measured by what the state produces (its outputs and outcomes, like in health and education). Rotberg and Boardman fall here. The second group, where Fukuyama falls, argues that these measures are too difficult for a variety of reasons and instead state capacity can best be measured by looking at how governments function, specifically bureaucratic procedures, capacity (in the sense of the ability to get things done) and autonomy (in the sense of protection from political micromanagement).

In the public sector unit at the World Bank, we fall solidly into the pro-Fukuyama camp, but come at it through a slightly different lens: public administration. 

The public sector can in general be disaggregated into two domains: upstream bodies at the center of government and downstream delivery bodies which deliver, commission or fund services under the policy direction of government. Both upstream and downstream bodies are allowed more or less autonomy from political control and/or micro-management. The argument is essentially whether we want to measure results downstream or upstream.

The “results” from the downstream bodies are of three types:

1. Services, such as health and education, housing, transport, electricity or security, through direct provision and through funding;
    
2. Management of infrastructure and other public investments which the private sector may be unable to finance or for which the private sector may be unwilling to bear all the risk; and
    
3. Regulation of social and economic behavior when necessary, such as food or road transport safety.

These are primarily the sector outputs (health, education, etc.) that Rotberg and Boardman (among others) want to measure. However, to know the quality of an output we need to know the strength of the causal relationship between the output and the achievement of the policy objective.  Although Atkinson thought that ultimately it was a feasible exercise, in his review of output measurement for the UK government he points out that the quality of an output, or service, is really a measure of “the attributable incremental contribution of the service to the outcome” ( Atkinson, Grice et al. 2005, p.42).  Causality in achieving public policy outcomes is notoriously hard to assign and thus quality of outputs is very hard to determine. 

For this reason, we agree with Fukuyama that it is best to measure the two types of results from central agencies:

1. Outcomes which are the product of their own capacity – including: ensuring that public revenues, expenditures and debt remain within agreed fiscal aggregates; maximizing cooperation between levels of government; developing and managing competing policy proposals;
    
2. The administrative procedures (design and enforcement of the rules of the game) that the downstream agencies must play by – interpretation of political priorities and translation into policy goals, allocation and management of public finances, creation and management of employment regimes, etc.

The next question, however, is how to measure these upstream results. This is where the Indicators of the Strength of Public Management Systems (ISPMS) project comes in. Together with several partners [1], the World Bank has launched ISPMS to identify and build consensus around a set of indicators to measure state capacity for the five main functions of central agencies: procurement, tax, public financial management, public administration and civil service, and public information. We have so far identified about 100 indicators that are being regularly collected and meet the following criteria:

• They are behavioral, capturing the functioning or performance of public institutions, to avoid the fashion trap of best practices which encourage mimicry of specific legal, organizational or institutional forms (Ashworth, G.Boyne et al. 2007). For example, the increasingly popular imposition of fiscal rules (numerical limits on the budgetary aggregates), driven by recent experiences of fiscal consolidation experiences (IMF 2010; Lassen 2010; OECD 2010), has at best a marginal impact on actual behaviors.
   
• They are “action-worthy”.  The literature is also replete with cases of central agencies driving real behavior changes in the public sector – but with little or no evidence that those changes matter.  For example, the debate about the value of New Public Management reforms for developing countries is primarily a discussion of the degree to which the central agencies should delegate flexibility over the use of inputs (staff, money, physical assets) to the downstream bodies in exchange for tighter accountability for results. Many of these reforms undoubtedly led to behavior change across the public sector – but whether it made any difference to service delivery or development outcomes is open to question (Schick 1998; Manning 2001).
   
• They reflect a clear concept of what they are measuring, or in other words, they are actionable. Indicators should be specific enough to point to clear policy actions that can be taken to change scores. Composite indicators that purport to measure the functioning or “effectiveness” of governments too frequently combine incongruous concepts. While they make for nice headlines and facilitate regression analysis, they provide little actual information that can be used by governments, practitioners and researchers to understand the true drivers of capacity and what can be done to effectuate change.
   
• Finally, data quality and comparability is paramount—without it, we won’t be able to draw any conclusions about why some countries perform better than others. Thus, the indicators should also be replicable. Results should be consistent across different assessors and the methodology transferable across cases and contexts.

Take a look at the dataset and let us know what you think. Is anything missing? We’re looking for new ideas. You can share yours at www.worldbank.org/ichallenge.
 

SOURCES:

• Ashworth, R., G.Boyne, et al. (2007), 'Escape from the Iron Cage? Organizational Change and Isomorphic Pressures in the Public Sector', Journal of Public Administration, Research and Theory, 19, 165-187.
• Atkinson, T., J. Grice, et al. (2005), Measurement of Government Output and Productivity for the National Accounts, Basingstoke, Palgrave.
• IMF (2010). Strategies for Fiscal Consolidation in the Post-Crisis World. IMF, Washington DC.
• Lassen, D. D. (2010). Fiscal Consolidations in Advanced Industrialized Democracies: Economics, Politics, and Governance. University of Copenhagen, Copenhagen.
• Manning, N. (2001), 'The Legacy of the New Public Management in Developing Countries', International Review of Administrative Sciences, 67 (2), 297-312.
• OECD (2010). Fiscal Consolidation: Requirements, Timing, Instruments and Institutional Arrangements. OECD, Paris.
• Schick, A. (1998), 'Why Most Developing Countries Should Not Try New Zealand's Reforms', World Bank Research Observer (International), 13, 23-31.