Government by numbers: How big a problem is gaming?

|

This page in:

Brian Talbots Flickr under creative commons

Governments are under pressure to show their ever more educated and informed citizens that schools, hospitals and other public services are getting better. Traditionally, they have done that by spending money and building things: look, a brand new hospital! Of course, everybody knows that there is more to service quality than dollars, bricks and mortar. But at least we can see and touch bricks and mortar.  How can we put a finger on service quality? 

To answer that question, governments have been turning to the technology of performance management: “key performance indicators” (KPIs) and measuring performance against them. Malaysia is one of the leaders in “government by numbers.” Its service delivery unit, PEMANDU, is proud of the reduction in crime, one of its own KPIs, over the last few years. But here we run into a problem which is posed in a recent blog which focuses on PEMANDU:
 
“Is data really a neutral arbiter? Data can be gamed and metrics can generate perverse incentives.  Since you tend to ‘get what you measure’, if the measure is flawed, it can drive public systems in very extreme ways, with negative consequences for service delivery.”
 
Sure enough, some of PEMANDU’s critics in Malaysia have alleged that the reduction is more apparent than real because the statistics have been manipulated. PEMANDU rejects that. However, any time a government manages a public agency on the basis of performance statistics which the agency itself is gathering – so that the agency is marking its own homework – it is leading the agency into temptation. And if the agency yields to the temptation and gets caught out, public confidence is sure to be undermined. 
 
The British police have been on the receiving end of allegations like those in Malaysia for almost twenty years. When the volume of the allegations became deafening about a year ago, culminating in the UK Statistics Authority withdrawing its triple-A rating from police crime figures, the government asked its police watchdog, Her Majesty’s Inspectorate of Constabulary (HMIC), to have a look.  Its report has just been published.
 
HMIC found no hard evidence of fiddling, but it did find that 21% of a self-selected survey group of 17,000 police staff said they had experienced pressure to misrecord crime data in the previous six months. This corroborates evidence that the practice has got so entrenched in the past that the police have coined their own nicknames for it:

  • "Cuffing": officers making crimes disappear by recording them as a "false report" or downgrading their seriousness.
  • "Stitching" (from "stitching up"): offenders being charged with a crime when there is insufficient evidence. Police may know that prosecutors will never proceed with the case, but it still appears in police records as a crime which has been "solved".
  • "Skewing": police activity being directed at easier-to-solve crimes, at the expense of more serious offences such as sex crimes
  • "Nodding" – where convicted offenders are persuaded to admit to crimes they have not committed, possibly in exchange for a lower sentence
The survey evidence has to be read carefully. 61% of the survey group said they had experienced no pressure.  Also, we don’t know how many of the 21% who had experienced pressure yielded to it, let alone to what extent crime data was distorted as a result. But it is clear that UK police forces’ crime data has not been a “neutral arbiter.” Even if the extent of fiddling could be shown to be minor, public confidence would still be undermined – and it has been: we can see that from the recent UK press coverage.
 
Stating the problem is the easy bit. What’s the solution? That is the subject of my second blog. Stay tuned! 

Authors

Willy McCourt

Senior public Sector Specialist

Sylvester Obong'o
December 17, 2014

Thank you very much for this incisive piece. The issue of performance measurement is gaining momentum in many countries because, performance measurement is considered modern. Driven by the famous saying- if you can't measure it, you can't improve, correct it, reward it and so on. The numbers have come in droves in many countries without corresponding improvement in public services.
I wish to state a few examples of cases I am most familiar with. During the structural adjustment credits to Kenya- In order to reduce the wage bill, the Government was required to retrench public servants. The World Bank and IMF set specific numbers. What the government did was to retrench a few but make up the numbers by abolishing positions in the establishment which had not been filled for years. Thus by analysing the in-post against establishment. The most absurd was however, when the president purpotedly reduced the size of government by merging ministries in 1997 without reducing the number of ministers.
However, the ranking of schools which has been going on sometime in Kenya produced one of the most subtle ways of gaming in order to achieve a higher ranking especially among private schools. The ranking of schools is based on the performance of the students in the school in the national exams. As a result it is a function of total number of students enrolled in a school. At the beginning some private schools deliberately decided to register few students. But because the private schools are also business oriented, some schools have devised ways of ensuring they still get the required fees but register other students in satelight schools where they enrol students they consider as weak, who will lower their mean grade. Thus a parent would take their child to School A which performed very well in exams, pay the exobitant fees, but come exam time, because that child was considered weak- and the school secretly registered them for exam under a satelight school, they are taken to those centers to sit for exams.
Lastly in the public service, after the introduction of performance contracting and the ranking of ministries, state corporations and local authorities, I have observed that because the ministries the agencies chose the areas they want to be measured on, most of the agencies insist on choosing targets which they can easily acheive and demonstrate and they have invested heavily in their reporting formats.
In both cases of course service delivery seems to have taken a backstage. Thus we are busy weighing the pig and losing our primary focus of the need to fatten the pig. As you said stating the problem is the easy bit- and I would wish to add part of the problem is the over emphasis being placed on measurements even on very basic systems whose structures cannot support a meaningful measurement regime often borrowed from the most developed systems.

Laurent Sauvage
December 18, 2014

Robust institutional governance, integrated management information system, self-control and independent control, triangulation of data, some redundancy of M&E mechanisms, traceability of operations, document management, audit trail, SMART indicators, balanced score card, comparability across borders and time, open access to data, fact-checking by media, by citizens, these are a few of my favorite things

Lanre Rotimi
December 20, 2014

Dear Laurent, thanks for your interesting comment. This is more of solutions. Can you please take a look at Willy's promised solutions on this link
http://blogs.worldbank.org/governance/government-numbers-how-can-we-sol…
and respond with focus on answers to anti gaming How questions, that includes How to implement with effective monitoring and evaluation of the implementation of the solutions you have suggested?

Dhirendra Krishna
December 17, 2014

Evaluation by the beneficiary of services provided by the Government is a more reliable performance index. Institutionalized social audit and action on findings on audit by citizens is needed. Evaluation at the cutting edge of administration by the citizens can lead to effective public accountability, instead of statistical evaluation by administrative hierarchy, that invariably lead to "fudged" reports.

Linn Hammergren
December 18, 2014

Well, yes, citizen evaluations should also be used, but often citizens don't know what to expect so using standard statistical indicators is also useful. For example, average caseloads can be compared across countries, and when a country (or court, or prosecutor's office) is at the very low end of the range, that is a sign that something is amiss. Also, while citizens see outcomes, there are many internal processes that can be tracked, giving a clue as to what might be going right or wrong.

R Srinivasan
December 18, 2014

With citizenry becoming more conscious of accountability in governance and 'governments' becoming aware that performance on social indicators hugely influence electoral preferences, it is but natural that there is pressure on governments to perform, tempting them sometimes to perform by 'manipulation', as the instances cited indicate. I think the way to minimize well meaning but misplaced enthusiasm on the part of public officials would be to devise monitoring mechanisms that include citizen reps, NGOs and media. That way, the temptation to fortify the statistics will be counter-balanced by the element of transparency.

Lanre Rotimi
December 18, 2014

Interesting article. We wait for your solutions before we share ours. However, the overarching point being made is that the problem is not just in Malaysia and UK alone but in each of the over 200 UN Member Countries and Non UN Member Countries in our world today. The difference is in the degree.

William McCourt
January 28, 2015

This is an umbrella reply to all the contributors. They have advanced the discussion– and they have educated me! I am attaching my reply to both my blogs, to make sure everyone sees it.
The fascinating detail from Kenya provided by Sylvester Obong’o shows that the problems we are discussing have been around for ages. In a way, this is encouraging, because if there are problems, there are some solutions, as contributors have shown. I have learned that there is a fair amount of agreement “out there” about how citizens get the benefits of government by numbers while keeping gaming to a minimum.
I am going to summarize the main points that I see in the discussion. Let’s start with a quotation from a classic paper by Donald Campbell which Charlie Undeland mentioned – it’s a bit chewy, but worth getting your head around:
“Worship of a quantitative indicator, leading both to a reduction in the validity of that indicator for its original purposes, and a corruption of the social processes it was designed to reflect … is one of the major problems to be solved if we are to achieve meaningful evaluations of our efforts at planned social change … We should study the social processes through which corruption (of the data) is being uncovered and try to design social systems that incorporate these features. In the Texarkana performance-contracting study, it was an “outside evaluator” who uncovered the problem. In a later U.S. performance-contracting study, the Seattle Teachers’ Union provided the watchdog role. We must seek out and institutionalize such objectivity-preserving features. We should also study the institutional form of those indicator systems, such as the census or the cost-of-living index in the U.S., which seem relatively immune to distortion.” (https://www.globalhivmeinfo.org/CapacityBuilding/Occasional%20Papers/08…)
* Triangulate data – i.e. have multiple -- the more the better -- external broad measures of quality that would feed into an overall assessment of the performance initiatives, reducing crime, improving exam results etc. as the case may be (Laurent Sauvage and Charlie Undeland)
* Among those measures, think about beneficiary evaluation (Dhirendra Krishna and R Srinivasan)
* Complement beneficiary evaluation with standard statistical indicators such as average caseloads for courts and prosecutor’s offices (Linn Hammergren)
I make no apology for ending on a corny note. If a public agency has the sincere intention of improving its performance, then government by numbers can indeed help. And if citizens are keeping an eye on what the public agency is doing, then the agency will have an incentive to be sincere. Service providers: purify your intention. Citizens: step up your scrutiny.

moviestarplanet Hack tool 2014
January 12, 2015

Children are far more protected on a dedicated video console system.
It's the eighth and last movie in the series based on the Love Arrives Softly novels by Janette Oke.
Total score is an indication of the overall player performance in game.