Given Jed's post last week on thinking through performance incentives for health workers, and the fact that the World Bank is in the throes of a reform process itself, a fascinating new paper from Imran Rasul and Daniel Rogger on autonomy and performance based incentives in Nigeria gives us some other food for thought. In a nutshell, Rasul and Rogger find that when management practices are more geared towards autonomy, project completion and quality improves. When they are related to performance based incentives, both project completion and quality are reduced. Clearly a result that is worthy of a bit more explanation...
For starters, Rasul and Rogger have quite a remarkable dataset. In 2006 and 2007, the Nigerian government tracked, at project level, the use and outcome of 10% of all federal social sector expenditure. This included visits to project sites by teams of engineers and civil society folks (not civil servants) who looked at the stage of completion of a project as well as the quality of the outcomes. And there were random checks by national teams and the Presidency to make sure these evaluations made sense. Rasul and Rogger then hand code all of this material ending up with data on 4721 projects (mostly construction projects such as boreholes, dams, buildings and roads) across 63 organizations, accounting for 8 percent of government spending or 800 million dollars. They then team up with a set of engineers to add variables on project complexity, and pull in data on technical specifications and the budget allocated to a given project. One thing to note that will be important for their analysis: projects of the same type (e.g. dams) are carried out by more than one organization in the same place.
Then, to get data on the bureaucracies themselves, Rasul and Rogger, use the work of Nick Bloom and John Van Reenen to develop an instrument in collaboration with folks in the Nigerian Presidency and Office of the Head of the Civil Service of the Federation. This survey was administered to managers (excluding the chief executive) in the 63 organizations and is designed to capture two main things. First, is autonomy, which they define broadly as the "the extent to which a civil service organization spreads the burden of the organization's work load, gives staff the opportunity and flexibility to achieve the organization's goals, the extent to which different tier bureaucrats can define organization policy, the flexibility of the organization's work culture, and the extent to which it is conducive to team work." The second aspect they are after is performance-based management, which they summarize as "the extent to which an organization collects indicators of project performance, how these indicators are reviewed, and whether bureaucrats are rewarded for achievements." Worthy of note here: these indices are positively correlated within organizations, on the order of .4-.5.
So what do they find? A one standard deviation increase in autonomy for bureaucrats is associated with a 14% increase in project completion, while a one standard deviation increase in performance-based management is associated with a 12% lower rate of project completion (for the record: 38% of projects were never started, and only 31% of projects are fully completed -- Rasul and Rogger use a continuous measure of completion). In terms of quality (here reduced to satisfactory or not), more autonomy is associated with more satisfactory quality, more performance-based management is associated with less.
Now management may choose to provide more autonomy for a range of reasons related to the project and the staff working on it for some good reasons. However, Rasul and Rogger find that the relationship between project completion and autonomy is fairly similar across projects of different size, complexity, structure of the organization implementing them, or the education level of the staff. On the performance-based management side, they find some more heterogeneity -- and heterogeneity which lines up with theory: performance-based incentives are associated with worse outcomes for more complex projects (which they use as a proxy for multi-tasking) and those that require cooperation among larger teams (proxied by the span of control of senior-tier folks).
Rasul and Rogger then go on to examine how these results are associated with the underlying characteristics of the bureaucrats themselves (yet another dataset). First, the longer the tenure of the bureaucrats, the more pronounced the basic result becomes -- more years of service translate into a higher positive correlation between autonomy and completion, and more a negative correlation between performance-based management and completion. Second, they find that the positive effect of autonomy for lower tier bureaucrats is reduced when senior management is more intrinsically motivated (defined here off of the reasons why they joined the civil service). They interpret this as the case of shifting autonomy, arguing it doesn't work as well for the lower tier folks when it is taken from the upper tier folks. They then move on to look at the effects of perceived corruption (as reported by the bureaucrats, and in no small measure either). Autonomy is associated with greater positive impacts when perceived corruption is higher, while the negative impacts of performance based incentives do not change.
To be sure, there is no experiment here (randomized or otherwise). So Rasul and Rogger do a fair bit of work to try to assure us that we aren't observing some wacky correlations. First, they tackle possible sorting by the bureaucrats themselves by providing some descriptive evidence that bureaucrats have limited autonomy over their assignments and they complement this with the fact that their bureaucrat characteristics are not correlated with management practices. Second, they take the same approach with the assignment of projects to organizations -- documenting the political process and also showing no correlation with management practices. Third, to address potential reverse causality, they use a SUR model to show that management practices don't seem to be driven by completion rates or project complexity. Lastly, to tackle possible omitted variable bias, they take apart the underlying components of the autonomy and performance indices to show separate impacts.
In the end, make of these relations what you will. But for sure this incredibly rich data and the analysis Rasul and Rogger do give us a lot of food to thought to complement what Jed had to say earlier and, more broadly, how we think about public services institutions and the way they are managed.
For starters, Rasul and Rogger have quite a remarkable dataset. In 2006 and 2007, the Nigerian government tracked, at project level, the use and outcome of 10% of all federal social sector expenditure. This included visits to project sites by teams of engineers and civil society folks (not civil servants) who looked at the stage of completion of a project as well as the quality of the outcomes. And there were random checks by national teams and the Presidency to make sure these evaluations made sense. Rasul and Rogger then hand code all of this material ending up with data on 4721 projects (mostly construction projects such as boreholes, dams, buildings and roads) across 63 organizations, accounting for 8 percent of government spending or 800 million dollars. They then team up with a set of engineers to add variables on project complexity, and pull in data on technical specifications and the budget allocated to a given project. One thing to note that will be important for their analysis: projects of the same type (e.g. dams) are carried out by more than one organization in the same place.
Then, to get data on the bureaucracies themselves, Rasul and Rogger, use the work of Nick Bloom and John Van Reenen to develop an instrument in collaboration with folks in the Nigerian Presidency and Office of the Head of the Civil Service of the Federation. This survey was administered to managers (excluding the chief executive) in the 63 organizations and is designed to capture two main things. First, is autonomy, which they define broadly as the "the extent to which a civil service organization spreads the burden of the organization's work load, gives staff the opportunity and flexibility to achieve the organization's goals, the extent to which different tier bureaucrats can define organization policy, the flexibility of the organization's work culture, and the extent to which it is conducive to team work." The second aspect they are after is performance-based management, which they summarize as "the extent to which an organization collects indicators of project performance, how these indicators are reviewed, and whether bureaucrats are rewarded for achievements." Worthy of note here: these indices are positively correlated within organizations, on the order of .4-.5.
So what do they find? A one standard deviation increase in autonomy for bureaucrats is associated with a 14% increase in project completion, while a one standard deviation increase in performance-based management is associated with a 12% lower rate of project completion (for the record: 38% of projects were never started, and only 31% of projects are fully completed -- Rasul and Rogger use a continuous measure of completion). In terms of quality (here reduced to satisfactory or not), more autonomy is associated with more satisfactory quality, more performance-based management is associated with less.
Now management may choose to provide more autonomy for a range of reasons related to the project and the staff working on it for some good reasons. However, Rasul and Rogger find that the relationship between project completion and autonomy is fairly similar across projects of different size, complexity, structure of the organization implementing them, or the education level of the staff. On the performance-based management side, they find some more heterogeneity -- and heterogeneity which lines up with theory: performance-based incentives are associated with worse outcomes for more complex projects (which they use as a proxy for multi-tasking) and those that require cooperation among larger teams (proxied by the span of control of senior-tier folks).
Rasul and Rogger then go on to examine how these results are associated with the underlying characteristics of the bureaucrats themselves (yet another dataset). First, the longer the tenure of the bureaucrats, the more pronounced the basic result becomes -- more years of service translate into a higher positive correlation between autonomy and completion, and more a negative correlation between performance-based management and completion. Second, they find that the positive effect of autonomy for lower tier bureaucrats is reduced when senior management is more intrinsically motivated (defined here off of the reasons why they joined the civil service). They interpret this as the case of shifting autonomy, arguing it doesn't work as well for the lower tier folks when it is taken from the upper tier folks. They then move on to look at the effects of perceived corruption (as reported by the bureaucrats, and in no small measure either). Autonomy is associated with greater positive impacts when perceived corruption is higher, while the negative impacts of performance based incentives do not change.
To be sure, there is no experiment here (randomized or otherwise). So Rasul and Rogger do a fair bit of work to try to assure us that we aren't observing some wacky correlations. First, they tackle possible sorting by the bureaucrats themselves by providing some descriptive evidence that bureaucrats have limited autonomy over their assignments and they complement this with the fact that their bureaucrat characteristics are not correlated with management practices. Second, they take the same approach with the assignment of projects to organizations -- documenting the political process and also showing no correlation with management practices. Third, to address potential reverse causality, they use a SUR model to show that management practices don't seem to be driven by completion rates or project complexity. Lastly, to tackle possible omitted variable bias, they take apart the underlying components of the autonomy and performance indices to show separate impacts.
In the end, make of these relations what you will. But for sure this incredibly rich data and the analysis Rasul and Rogger do give us a lot of food to thought to complement what Jed had to say earlier and, more broadly, how we think about public services institutions and the way they are managed.
Join the Conversation