Mystery clients in development research

When I turned 16, I got my first formal sector job at  a movie theater. The theater was part of a large chain, and occasionally the head office would send mystery shoppers -- employees posing as moviegoers -- to test the quality of our customer service. A colleague of mine was put on probation for getting impatient with an elderly mystery shopper at the ticket window.

Mystery shoppers are not merely an instrument of corporate chains. They can also be an illuminating data collection strategy in development research. They go by different names: mystery clients, mystery shoppers, covert shoppers, simulated patients, and standardized patients are all versions of this phenomenon. They also aren't new. Twenty-five to thirty years ago, studies were using mystery clients to evaluate family planning programs in  Ghana,  Morocco, and  Haiti.

 

Below are some of the main ways that this tool has been used in recent development research:

 

1. Testing for substandard drugs

 

Some of these studies are merely descriptive: In a 2010 study,  Bate et al. sent covert shoppers into private pharmacies in 11 African cities and 8 cities in non-African middle income countries to buy essential medicines for malaria, tuberculosis, and bacterial infections. The study then analyzed which medicines were unregistered and how many failed quality tests. (On the latter, failure ranged from 0% in Istanbul to 32% in Kampala). A different  Bate et al. study in 2011 reported different outcomes for a slightly different set of cities.

 

Others impose experimental variation. For example,  Fitzpatrick (2015) randomized the script that covert shoppers recited to drug vendors in Uganda to test how much patient knowledge about their likely diagnosis and the appropriate treatment affected the price and quality of the drugs they were sold. Shoppers with more information received lower prices but -- surprisingly -- were more likely to receive counterfeit drugs! (This paper was summarized in detail on  Development Impact a few months back.)

 

2. Testing the quality of medical care and advice

 

Mystery shoppers can highlight the difference between provider knowledge (which can be gleaned from questionnaires and vignettes) and actual practice.  Friedman et al. (2015) used a mobile phone text message campaign to provide follow-up training to pharmacists in Ghana on the appropriate treatment for childhood diarrhea. They then combine data from face-to-face interviews and mystery client visits to demonstrate that the program improved self-report practices but NOT actual practices. (Cue  sad trombone sound.)

 

Das et al. (2012) sent standardized patients to private and public primary health care facilities in India and found that "correct diagnoses were rare" and "incorrect treatments were widely prescribed."  Mohanan et al. (2015) combined vignettes with standardized patients and found a pronounced gap between knowledge and action in rural Bihar, India.  Another standardized patient project is ongoing in Kenya.

 

With some parallels to the Fitzpatrick study above,  Currie et al. 2011 reports on sending simulated patients to physicians in China with instructions to describe flu-like symptoms. Some patients were randomly assigned to also say, "I learned from the internet that simple flu/cold patients should not take antibiotics." (I'm not sure how they pronounced the "/".) The result was fewer antibiotic prescriptions, fewer drug expenditures, more information on drug side effects, and - here's the clincher - ruder doctors. (So I guess you have to choose: Friendly doctor or appropriate prescription. Can't we have both?)

 

Mystery shoppers were sent both to general shops and to drug shops in Tanzania to test treatment and referral policies in  Hetzel et al. 2008. General shops are the first place that 25-30% of people go with a fever, and their treatments were substandard.

 

3. Testing for corruption

 

Dizon-Ross et al. 2014 sent mystery clients to test for mistargeting and corruption in programs distributing free bed nets to pregnant women in Ghana, Kenya, and Uganda. Male mystery clients -- who were visibly not eligible -- were sent to distribution centers to try and obtain a subsidized bed net (by any means necessary!). Across the three countries, fewer than 5% of mystery client attempts were successful in getting a net.

 

4. What about outside of health?

 

Most notably, a brief search turned up little outside of the health sector: This seems like a missed opportunity (although mystery shoppers  were used to  test the competency of librarians in Kenya, so that's something).

 

One article mentions the use of mystery shoppers to monitor the performance of workers in one-stop government centers in Georgia, but there is no research reported. (One-stop government centers seek to make legal processes easier by offering many government services in one place.) The growing field of impact evaluations in governance (see  here and  here, for example) might be a natural space to use this tool, testing the speed and quality of service provision.

 

Where else are researchers already using mystery shoppers in development research, or where else should they be?

Update: A couple of readers immediately pointed me to examples of exactly what I proposed above!