Statistics about statistics: How do we measure statistical performance?

Why statistics about statistics are vital

Statistics are at the core of modern decision-making. They tell policymakers how many people are in the hospital, how the economy is doing, and by how much CO2 emissions are ascending. In the past year, their importance rose as official (and, at times, unofficial) statistics were deployed to inform a vast array of COVID-19 measures, as well as economic and social recovery programmes. While many advanced economies did comparatively well in using data for decision-making, a significant number of low- and middle-income countries struggled to collect data or make sense of existing data (see Misra, Schmidt and Harrison 2020).

Assessing the quality of data during a crisis is not an easy task. To know how good a country’s data and statistics are, one needs to understand how well the system is doing that generates them.  For instance, how good is the infrastructure in a statistical office? Do the people that work there have the right skills? What type of data are produced using digital tools such as tablets? And, most importantly, how are the data used that the government produces? Thus, evaluating statistical performance quickly turns into a complex analysis of governance mechanisms, institutions, and people.
 

High ambitions – transitioning from capacity to performance

Amidst an ever-changing data landscape, statistical capacity metrics become outdated soon after they’re conceived. For years, we have been measuring statistical capacity on the basis of how many data were available. With the invention of the Statistical Capacity Index (SCI) by the World Bank, we started looking at a wider picture, taking into account the periodicity and methodological soundness of the data products in the public space. Yet the advent of the data revolution, coupled with the disruptive expansion of the data ecosystem with innovative, unstructured data sources and new actors and data users, urged a new approach to capacity measurement. New definitions of ‘capacity’ go beyond technical skills to include legal frameworks, coordination with other data providers and leadership skills (see PARIS21 approach on “Capacity Development 4.0” (CD4.0)).

But given the pace of digital transformation, systems and their capacities change rapidly. These developments not only affect how statistics are generated, but also how they are used. 

The Statistical Performance Indicators (SPI) developed by the World Bank are a breakthrough in assessing the performance of those institutions that measure our economic and social reality. Displaying twenty-two dimensions in five pillars, the framework provides an ambitious sketch of a modern statistical system across high- and low-income countries, and it links to the 2030 Agenda for Sustainable Development. The SPI is finally advancing a further attempt in measuring what matters most in a coherent way.

Despite these novelties, the framework remains a starting point on a journey to a complete statistical picture of ground truth. A first potential area of improvement relates to extending country coverage. As existing work has shown, data sharing and use by civil society, academia and the private sector varies widely, especially in low-income and fragile states. For this reason, these concepts cannot be measured on a global scale yet since harmonized and methodology-proof indicators do not exist.

Second, some dimensions touch upon the new roles of the national statistical office as a data steward. These indicators refer to “effectiveness of advisory and analytical services” as well as “availability and use of data services” without describing how first best data collection efforts or a methodology should look. This approach may disincentivize partners to step up as “indicator owners” and to propose data and methodologies that, although not yet precisely targeting the dimensions of the SPI, could still serve statistical performance measurement.

Third, benchmarking countries against an index alone will not do the trick in boosting statistical performance. Resources, IT infrastructure and skills are missing across a wide range of low-income countries. Meeting the needs of diverse countries requires huge investments across all dimensions measured by the SPI framework. The more partners use the index and work towards reporting accurately, the better development providers can allocate financial resources, technical assistance and knowledge to those institutions that need it most.
 

Filling the gaps – working together towards robust new measurements

The SPI calls for more collaboration to fill the measurement gaps in statistical performance. The statistical community has a joint responsibility to push the measurement agenda further. PARIS21 has taken on that challenge and has started to identify priority areas within the framework.

One of these measurement areas is financing for data and statistics. Financial resources are crucial for scaling the performance of a national statistical system. This message is manifest in the global action plan of the High Level Group on Partnership, Coordination and Capacity-Building for statistics for the 2030 Agenda for Sustainable Development (HLG-PCCB), which calls for policy leaders to achieve a global pact or alliance that recognises that funding modernisation efforts of national statistical offices is essential to the achievement of the 2030 Agenda. As the secretariat of the Bern Network, a multi-stakeholder alliance for more and better development data financing, PARIS21 has launched the Clearinghouse for Financing Development Data at the UN World Data Forum in October. The platform which was developed by PARIS21 and ODW with valuable input from the World Bank, the United Nations Statistics Division and other Bern Network Members (accessible at www.smartdatafinance.org) is a one-stop-shop to align the priorities of aid providers and recipients, and will help them identify new partnerships, bring projects to scale, and build a stronger case for data and statistics (see Figure 1). To reach this goal, the platform displays project-level data, strategic sector-level funding and SDG-relevant flows for IDA countries. Paris21 is still working to identify and crowd relevant project-level data onto the platform, and so the Clearinghouse remains a work in progress. The data published on the platform – now and in the future – will help inform decision-making for the Global Data Facility, a new, World Bank-hosted fund to support data and statistics priorities at the global, regional, national, and community levels. The data provided by the SPI and the Clearinghouse can contribute to mobilize and coordinate donor support for data and statistics priorities, to catalyze further financing, including World Bank IDA/IBRD financing, and enable long-term support for priorities by strengthening domestic investments in data and statistics.

Figure 1: Preview of the Clearinghouse for Financing Development Data

A second area refers to coordination and governance in the statistical system. In 2020, PARIS21 and the Philippine Statistical Authority convened a high-level task team to unravel the complex web of coordination mechanisms in statistical systems and measure their effectiveness. Experts grappled with the question of how to capture the different coordination mechanisms applied across countries (see Figure 2). Colombia, for instance, introduced a pioneering data stewardship strategy and cross-sectoral working groups as key instruments of statistical coordination. Grenada, on the other hand, had just designed their statistical law and concluded a multi-year statistical plan. Discussions resulted in a set of five indicators aimed at measuring coordination capacity at the global level. Defining indicators precise and objective enough to capture statistical performance while allowing for sufficient flexibility will be crucial in finding common methodologies for SPI dimensions such as “data services”.
 

Figure 2: A toolbox of coordination mechanisms and associated coordination capabilities

Third, legal frameworks are foundational in steering a data ecosystem. While current indicators measure the existence of statistical laws (see UNSD SDG Indicator 17.18.2), less is known about the quality of the laws in place, and the ways they are implemented. PARIS21 plans an ambitious project on digitising statistical laws – starting with a small pilot to understand legal frameworks and changes induced by new data actors. The project aims to parse through legal texts to grasp their quality and shed light on data sharing and privacy protection regulations. Insights might help develop global indicators on various criteria of statistical laws aligned with the challenges of a modern data ecosystem.
 

Way forward

Statistical systems operate in a dynamic environment. Especially in uncertain times, they have to navigate by sight, grounded in statistics-driven decisions.  Building on the current SPI framework, we see three avenues for future collaboration:

1. Improve the data collection pipeline: Measuring statistical performance requires countries to provide their data. While incentives such as the better allocation of financial resources exist, the reporting should be anchored in a trusted relationship and not create an additional burden. An optimal data collection method would thus entail both data reporting and data validation. One way to do so is decentralised reporting hubs that establish a lasting data infrastructure, new skills and long-term institutional change in the partner countries.

2. Encourage country ownership and acceptance: Partner countries should themselves be encouraged to develop methodologies that capture the on-the-ground reality. Striking the right balance between a statistical community-driven approach and a country-driven approach can help designing better indicators. A combination of teams and pilot studies aimed at the specific dimensions in the SPI would guarantee a 360-degree view going beyond inputs and output and aiming at the measurement of outcomes.

3. Learn from what works to accelerate progress: The modern evolving data ecosystem requires regular validation and review of existing measurements. Regular events discussing meta-measurements of the statistical community will be fundamental to exchange knowledge, best practices and lessons learned amongst statisticians, donors and development partners and create long-lasting global partnerships.

Together with the World Bank, PARIS21 looks forward to move forward on all three of the pathways outlined above to improve statistics about statistics. This in turn, might help to boost statistical performance to next levels, and modernize statistical systems in low and middle-income countries.