Syndicate content

Mystery clients in development research

David Evans's picture
When I turned 16, I got my first formal sector job at a movie theater. The theater was part of a large chain, and occasionally the head office would send mystery shoppers -- employees posing as moviegoers -- to test the quality of our customer service. A colleague of mine was put on probation for getting impatient with an elderly mystery shopper at the ticket window.
Mystery shoppers are not merely an instrument of corporate chains. They can also be an illuminating data collection strategy in development research. They go by different names: mystery clients, mystery shoppers, covert shoppers, simulated patients, and standardized patients are all versions of this phenomenon. They also aren't new. Twenty-five to thirty years ago, studies were using mystery clients to evaluate family planning programs in GhanaMorocco, and Haiti.
 
Below are some of the main ways that this tool has been used in recent development research:
 
1. Testing for substandard drugs
 
Some of these studies are merely descriptive: In a 2010 study, Bate et al. sent covert shoppers into private pharmacies in 11 African cities and 8 cities in non-African middle income countries to buy essential medicines for malaria, tuberculosis, and bacterial infections. The study then analyzed which medicines were unregistered and how many failed quality tests. (On the latter, failure ranged from 0% in Istanbul to 32% in Kampala). A different Bate et al. study in 2011 reported different outcomes for a slightly different set of cities.
 
Others impose experimental variation. For example, Fitzpatrick (2015) randomized the script that covert shoppers recited to drug vendors in Uganda to test how much patient knowledge about their likely diagnosis and the appropriate treatment affected the price and quality of the drugs they were sold. Shoppers with more information received lower prices but -- surprisingly -- were more likely to receive counterfeit drugs! (This paper was summarized in detail on Development Impact a few months back.)
 
2. Testing the quality of medical care and advice
 
Mystery shoppers can highlight the difference between provider knowledge (which can be gleaned from questionnaires and vignettes) and actual practice. Friedman et al. (2015) used a mobile phone text message campaign to provide follow-up training to pharmacists in Ghana on the appropriate treatment for childhood diarrhea. They then combine data from face-to-face interviews and mystery client visits to demonstrate that the program improved self-report practices but NOT actual practices. (Cue sad trombone sound.)
 
Das et al. (2012) sent standardized patients to private and public primary health care facilities in India and found that "correct diagnoses were rare" and "incorrect treatments were widely prescribed." Mohanan et al. (2015) combined vignettes with standardized patients and found a pronounced gap between knowledge and action in rural Bihar, India. Another standardized patient project is ongoing in Kenya.
 
With some parallels to the Fitzpatrick study above, Currie et al. 2011 reports on sending simulated patients to physicians in China with instructions to describe flu-like symptoms. Some patients were randomly assigned to also say, "I learned from the internet that simple flu/cold patients should not take antibiotics." (I'm not sure how they pronounced the "/".) The result was fewer antibiotic prescriptions, fewer drug expenditures, more information on drug side effects, and - here's the clincher - ruder doctors. (So I guess you have to choose: Friendly doctor or appropriate prescription. Can't we have both?)
 
Mystery shoppers were sent both to general shops and to drug shops in Tanzania to test treatment and referral policies in Hetzel et al. 2008. General shops are the first place that 25-30% of people go with a fever, and their treatments were substandard.
 
3. Testing for corruption
 
Dizon-Ross et al. 2014 sent mystery clients to test for mistargeting and corruption in programs distributing free bed nets to pregnant women in Ghana, Kenya, and Uganda. Male mystery clients -- who were visibly not eligible -- were sent to distribution centers to try and obtain a subsidized bed net (by any means necessary!). Across the three countries, fewer than 5% of mystery client attempts were successful in getting a net.
 
4. What about outside of health?
 
Most notably, a brief search turned up little outside of the health sector: This seems like a missed opportunity (although mystery shoppers were used to test the competency of librarians in Kenya, so that's something).
 
One article mentions the use of mystery shoppers to monitor the performance of workers in one-stop government centers in Georgia, but there is no research reported. (One-stop government centers seek to make legal processes easier by offering many government services in one place.) The growing field of impact evaluations in governance (see here and here, for example) might be a natural space to use this tool, testing the speed and quality of service provision.
 
Where else are researchers already using mystery shoppers in development research, or where else should they be?

Update: A couple of readers immediately pointed me to examples of exactly what I proposed above!
  • Banerjee et al. evaluate four potential interventions to improve police performance in Rajasthan, India. They then use decoy visits to the police station to test. "Interestingly, the visits of decoy surveyors pretending to register crimes at police stations, which were initially conceived of primarily as a method of data collection, also had an effect of their own on police behavior--suggesting that a simple increase in monitoring may have a positive effect on police performance."
  • Bertrand et al. (2007) doesn't use the phrase "mystery clients" or "decoy visits," but they offer a bonus to individuals to obtain a driver's license in India and find that the individuals are likely to get a license but not to pass the driver's test and yes to pay a bribe.
  • Gine et al. (2014) use mystery clients to evaluate services as financial institutions in Mexico. Researchers varied the financial sophistication and appearance of the client, among other variables. "Staff voluntarily provides little information about avoidable fees, especially to auditors trained to reveal little knowledge about the market. In addition, clients are almost never offered the cheapest product, most likely because staff is incentivized to offer more expensive products that are thus more profitable to the institution."

Comments

Submitted by Gautam Bastian on

Life Insurance in India: Anagol, Santosh, Shawn Cole, and Shayak Sarkar. "Understanding the Advice of Commissions-Motivated Agents: Evidence from the Indian Life Insurance Market." Harvard Business School Working Paper, No. 12-055, January 2012. (Revised March 2013.)

Submitted by Alex Erskine on

You might also be impressed - I am - by the mystery shopping exercise conducted by Sharman in https://www.griffith.edu.au/__data/assets/pdf_file/0008/454625/Oct2012-Global-Shell-Games.Media-Summary.10Oct12.pdf
I think this approach is very practical for testing compliance in several regulatory settings. ASIC uses mystery shopping for financial advice and insurance sales to test quality - see examples at http://asic.gov.au/regulatory-resources/find-a-document/reports/rep-279-shadow-shopping-study-of-retirement-advice/ and http://download.asic.gov.au/media/1343720/rep256-issued-19-October-2011.pdf

Add new comment