Politics and Governance: calling for evaluation of “atypical” interventions: Guest Blog by Stuti Khemani

A “meta” problem facing not only impact evaluation work but all development policy dialogue is perverse behavior in the public sector to not pursue evidence-based, technically sound policies. Politics and governance come between statistically significant research results and real impact in the world. We confront these problems in a policy research report that has been described as having transformational implications for the business of international development assistance. And we derive implications for a research agenda that involves atypical impact evaluations that would complement work on how to fix the pipes with work on how to fix the institutions that would fix the pipes.

Consider the following situation that happens all too often: RCTs identify interventions that “work”, but these are either not incorporated into policy at all, and even when they are, they fail to be effectively implemented. How do we deal with the devil, as Dhaliwal and Hanna describe it, to get policies implemented even when they are taken-up? We can bypass this problem by side-stepping the public sector altogether, such as by giving directly, for example, but that does not make it go away. The idea of the role of the state in providing the public goods and security needed for sustainable development is resurgent. If the state cannot or should not be bypassed, we need to build knowledge through impact evaluation of how to make politics and governance work for development, not against it.

We pull out this single thread from a complex web of political economy literature: that the selection and sanctioning of leaders is the key to institutions. Building better institutions requires that the leaders who wield power within those institutions are selected and sanctioned on the basis of performance in pursuing the best known solutions to public good problems. Evaluation is needed of interventions that enable good “types” of leaders to emerge, and that strengthen incentives of all types of leaders to pursue policies (fixing pipes) on the basis of assessing and debating the best available technical knowledge (of how to fix pipes). Trends in how leaders are currently being selected and sanctioned around the world suggest that the local level within countries across the political spectrum is ripe for such evaluation.

The following figure we uncovered by plotting countries according to their Polity IV score on the extent to which they are “democratic”, shows that the overall distribution of political institutions across countries has steadily shifted toward those that allow space for greater political engagement by citizens in selecting and sanctioning the leaders who wield power in government. Although some individual countries have experienced reversals to more autocratic institutions or seen little change, overall the trend has been toward greater opportunities for political engagement.
Source: Data from the Polity IV project. The Polity IV Score is a measure of state authority varying on a 21-point scale ranging from −10 (which corresponds to hereditary monarchy) to +10 (which corresponds to the Polity IV view of consolidated democracy). Higher values are thus associated with more democratic institutions.

The dramatic spread of elections at national and at local levels, even within countries with authoritarian national political institutions, has created unprecedented opportunities for citizens to influence the selection and incentives of elites. Citizens are engaging in the political process as individual voters, and as contenders for political office. Even in countries where the national political system is closed to competition, there exists a local level that is open to the entry of new leaders or political entrepreneurs.

But, of course, elections are hardly enough. Elections can yield populist demagogues who gain power by exploiting divisions among citizens. Elections can be purchased through money and violence, allowing leaders to get away with stealing private rents from public office. We discuss the evidence for this alongside the evidence of electoral accountability, highlighting the nuances of how specific characteristics of electoral contestation—whether or not it revolves around public good issues versus redistributive conflict—matters for outcomes.

Elections are also not enough because we need institutions that govern the day to day functioning of a myriad agencies within government. But we show in the report that elections cast a long shadow on these day-to-day institutions through the leaders that are selected and the behavioral norms in the public sector that these leaders shape. 

We need more knowledge about what types of citizens enter as political entrepreneurs for leadership positions (selection of leaders); and, what determines the issues that citizens choose when deciding how to sanction or reward leaders (which determines incentives of leaders). Both selection and incentives of leaders are influenced by the distribution of preferences among citizens over public policies—what different citizens think the state should be doing. We need more knowledge about how media, communication and information shape the distribution of these preferences, over and above how information matters for accountability. We need more evidence to test our main argument: that shifting the basis for leader selection and sanctioning will ultimately contribute to improving service delivery and development outcomes across sectors. More healthy and educated children; more clean water; more navigable roads; greater private investments and income growth; less corruption; less violent conflict; etcetera.

So, here’s our proposed road map for the research community out there. Identify the “critical” jurisdictions within any given country that have the following properties, as the “last mile” of service delivery and the “first mile” for selecting leaders:

• Local political contestation for leadership positions, such as through elections
• Concrete management powers over public sector workers and public spending programs

The trade-off in identifying these jurisdictions and the research questions is going to be between finding a “low-enough” level at which causal inference is possible (say, through randomization of an intervention or a discontinuity design) and sample size is affordable (eg. villages) versus a “high-enough” level where leaders have real powers of management over public resources (eg. municipalities with taxation authority).

• Innovate survey instruments to gather data on “types” of leaders contesting in these jurisdictions (not just the incumbent, but actual and potential contenders)
• Innovate survey instruments, or create lab-in-the-field experiments to measure the distribution of preferences over public policies
• Work with policy actors (governments, first and foremost, but also NGOs and external aid agencies) to design interventions aimed at improving the quality of leaders and shifting preferences towards public goods (eg. media campaign to increase demand for public health)
• Evaluate impact on development outcomes, and design it so you can tell us something about the mechanisms of impact

We are going to use our report’s dissemination platform to try to generate demand and entry points for this type of research. This may create opportunities for impact evaluations at “meta-levels” such as districts, municipalities or provinces which may require very large samples. So stay tuned. We hope you will engage and contribute to this agenda.