Government by numbers: How can we solve the gaming problem?

|

This page in:

Riccardo Ghinelli under creative commons

In my last blog post, I showed that while governments are increasingly using the technology to demonstrate that services are improving, their efforts risk being undermined by “gaming” – in other words, fiddling the performance statistics to make things look better than they really are.
 
We focused on the problem in the UK.  In this blog, I look at what the UK has and hasn’t done to address the problem, and what we can learn from that. 

The first thing that the UK did to address gaming was to create an Office of National Statistics on an independent footing, reporting to Parliament rather than the Executive. This was not a response to the problems with crime data, but to broader complaints going all the way back to the 1980s that official statistics were being manipulated.  For example, the definition of unemployment changed 29 times during the 1980s. The suspicion that some of the changes were politically driven was hard to avoid.
 
The second thing was to give the UK statistics body a role in regulating the statistics that public agencies produce. (We saw in my earlier blog post how the Authority withdrew its triple-A rating from police crime data.) 
 
The third thing – again, not a specific response the crime data problem - is that every year since 1981, there has been an independent crime survey, based on citizens’ self-reports. It is regarded as more reliable than police data, because it includes crimes which citizens suffer but don’t report to the police (for example, sex offences are underreported), and because reports are not filtered by the police. 
 
After that, the report into crime data by the police watchdog, HMIC, which we discussed in the previous blog, made thirteen recommendations, which include training for police officers on how to record crimes accurately, and the advice that performance targets, personal performance appraisals, and decisions about remuneration or promotion should no longer be based on police-recorded crime data.
 
How have the UK police responded to the HMIC and other criticisms? One telling statistic is that although crime is reported to have dropped in the latest annual independent crime survey, it is actually reported to have increased in the police’s own data. It appears that the police have already taken steps to record crimes that they used to leave out. This does not mean that the problem is wholly solved, however. Critics have suggested that even the National Crime Survey does not give an accurate picture.  Crime may not have dropped as much as the statistics suggest, because criminals may have migrated from crimes like burglary to the more lucrative pastures of cybercrime, which even the National Crime Survey admits that it has not found a way of capturing so far. 
 
What do we learn from all this? 

  1. Service performance is not a neutral arbiter.  In the UK, just as in Malaysia and the US, management by numbers has created a perverse incentive to fiddle the statistics which risks undermining the entire initiative. Any government that goes down this road without taking steps to minimize gaming is acquiescing in data fiddling. It is also shooting itself in the foot, because it is undermining the public confidence which management by numbers is supposed to increase. 
  2. The steps needed are measures to ensure the integrity of the statistics. Government statistics agencies should be independent, and should be able to regulate the statistics gathered by line agencies. 
Sceptics may question the realism of those steps. Won’t managers always favor police officers and other service providers who deliver results, so that the perverse incentive will still be there? Yes they will, and yes it will - there is no watertight solution; but, arguably, no more than in other areas of public management, such as where a central bank has independence to set interest rates free from political interference; or, to take an example closer to home, an agency like the World Bank sets up an internal evaluation function on an independent footing.


Management by numbers is not the only way to improve public services. Competition between service providers, and good old-fashioned professional upgrading through training etc. and other approaches have their place. But the final message of the Malaysian, American and British experience is that governments that go down the numbers road to improve service performance will lose public confidence unless they recognize that there will be gaming, and unless they stamp on it.​
 

Are there other exampels we can learn from? Tell us in the comments. 

Authors

Willy McCourt

Senior public Sector Specialist

Lanre Rotimi
December 19, 2014

Thanks for the solutions you have shared. Are these really solutions? If the UK Government does not know why the authorities in its various agencies fiddling with statistics, is fighting the UK Society, they will not know how to fight the authorities in its various agencies fiddling with statistics.
You have used the UK Police as Case Study. You have used exotic names to describe Corruption and you have provided evidence that the UK Government is fighting this type of Corruption with Kid Gloves hence there is no sanction of any type. No wonder the disease cannot be cured in past and present UK Society.
There is a high probability that each agency of each UK Government will be found to have more or less the same poor scorecard as the UK Police if subject to the same level of scrutiny with regard to fiddling with statistics. And this underlines the need to give serious attention to serious issues of serious business raised in your two blog posts, if the UK Government genuinely desires to achieve increasing convergence between its National Development Cooperation and International Development Cooperation Vision Intention and Reality.
As stated in comment on earlier blog post, the problem is on Developed Countries - UK, US, France, Germany, Russia etc, Developing Countries - Nigeria, South Africa, China, India, Brazil etc and International Institutions - World Bank Group, IMF, UN, EC etc sides and not just Countries like UK, US and Malaysia and International Institutions like World Bank Group specifically mentioned in your two blog posts.
If you are really serious about finding practical solutions, fully implementing these solutions and effectively monitoring and evaluating the implementation, we are glad to exchange ideas on ACTION Next Steps Moving Forward. The Ball is in your Court. Will you Play?......

William McCourt
January 28, 2015

This is an umbrella reply to all the contributors. They have advanced the discussion– and they have educated me! I am attaching my reply to both my blogs, to make sure everyone sees it.
The fascinating detail from Kenya provided by Sylvester Obong’o shows that the problems we are discussing have been around for ages. In a way, this is encouraging, because if there are problems, there are some solutions, as contributors have shown. I have learned that there is a fair amount of agreement “out there” about how citizens get the benefits of government by numbers while keeping gaming to a minimum.
I am going to summarize the main points that I see in the discussion. Let’s start with a quotation from a classic paper by Donald Campbell which Charlie Undeland mentioned – it’s a bit chewy, but worth getting your head around:
“Worship of a quantitative indicator, leading both to a reduction in the validity of that indicator for its original purposes, and a corruption of the social processes it was designed to reflect … is one of the major problems to be solved if we are to achieve meaningful evaluations of our efforts at planned social change … We should study the social processes through which corruption (of the data) is being uncovered and try to design social systems that incorporate these features. In the Texarkana performance-contracting study, it was an “outside evaluator” who uncovered the problem. In a later U.S. performance-contracting study, the Seattle Teachers’ Union provided the watchdog role. We must seek out and institutionalize such objectivity-preserving features. We should also study the institutional form of those indicator systems, such as the census or the cost-of-living index in the U.S., which seem relatively immune to distortion.” (https://www.globalhivmeinfo.org/CapacityBuilding/Occasional%20Papers/08…)
* Triangulate data – i.e. have multiple -- the more the better -- external broad measures of quality that would feed into an overall assessment of the performance initiatives, reducing crime, improving exam results etc. as the case may be (Laurent Sauvage and Charlie Undeland)
* Among those measures, think about beneficiary evaluation (Dhirendra Krishna and R Srinivasan)
* Complement beneficiary evaluation with standard statistical indicators such as average caseloads for courts and prosecutor’s offices (Linn Hammergren)
I make no apology for ending on a corny note. If a public agency has the sincere intention of improving its performance, then government by numbers can indeed help. And if citizens are keeping an eye on what the public agency is doing, then the agency will have an incentive to be sincere. Service providers: purify your intention. Citizens: step up your scrutiny.

Charlie Undeland
January 07, 2015

A big thank you to Willy for pointing out the paper and raising these interesting issues.
I think gaming is near sure to happen even if it can (and should) be reduced. There are lots of examples of gaming and literature beginning I think with 'Campbell's Law' about why this occurs. Need only think about the scandal in Atlanta over the effects of performance mgmt in the No Child Left Behind Act, or, closer to home, Natsios's entertaining critique of development agencies' emphasis on measures and the problems with incentives it creates.
Seems to me that measuring the impact of measuring is at least as important for credibility than guarding against fiddling with the numbers. Do the effects of gaming outweigh the benefits that plausibly are influenced by use of performance measures? Here there are the same issues for measuring these effects(what is measured, who is doing it, identifying causality, etc.). But it seems to me that as an additional control on having specific performance measures be effective one would want multiple -- the more the better -- external broad measures of quality that would feed into an overall assessment. This would especially be the case before and after performance measures are introduced or enhanced. So for the UK case, is there higher citizen satisfaction in govt. efforts on crime, the crime survey, other reporting on crime, over time given that performance measures are used more? The combined assessments would be as important for validation than more robust, ahem, policing of measurements.