How to use mystery clients in development research

May 5, 2015

David Evans

Senior Economist, World Bank

When I turned 16, I got my first formal sector job at a movie theater. The theater was part of a large chain, and occasionally the head office would send mystery shoppers — employees posing as moviegoers — to test the quality of our customer service. A colleague of mine was put on probation for getting impatient with an elderly mystery shopper at the ticket window.

Mystery shoppers are not merely an instrument of corporate chains. They can also be an illuminating data collection strategy in development research. They go by different names: mystery clients, mystery shoppers, covert shoppers, simulated patients, and standardized patients are all versions of this phenomenon. They also aren’t new. Twenty-five to thirty years ago, studies were using mystery clients to evaluate family planning programs in Ghana, Morocco, and Haiti.

Below are some of the main ways that this tool has been used in recent development research:

1. Testing for substandard drugs

Some of these studies are merely descriptive: In a 2010 study, Bate et al. sent covert shoppers into private pharmacies in 11 African cities and 8 cities in non-African middle income countries to buy essential medicines for malaria, tuberculosis, and bacterial infections. The study then analyzed which medicines were unregistered and how many failed quality tests. (On the latter, failure ranged from 0% in Istanbul to 32% in Kampala). A different Bate et al. study in 2011 reported different outcomes for a slightly different set of cities.

Others impose experimental variation. For example, Fitzpatrick (2015) randomized the script that covert shoppers recited to drug vendors in Uganda to test how much patient knowledge about their likely diagnosis and the appropriate treatment affected the price and quality of the drugs they were sold. Shoppers with more information received lower prices but — surprisingly — were more likely to receive counterfeit drugs! (This paper was summarized in detail on Development Impact a few months back.)

2. Testing the quality of medical care and advice

Mystery shoppers can highlight the difference between provider knowledge (which can be gleaned from questionnaires and vignettes) and actual practice. Friedman et al. (2015) used a mobile phone text message campaign to provide follow-up training to pharmacists in Ghana on the appropriate treatment for childhood diarrhea. They then combine data from face-to-face interviews and mystery client visits to demonstrate that the program improved self-report practices but NOT actual practices. (Cue sad trombone sound.)

Das et al. (2012) sent standardized patients to private and public primary health care facilities in India and found that “correct diagnoses were rare” and “incorrect treatments were widely prescribed.” Mohanan et al. (2015) combined vignettes with standardized patients and found a pronounced gap between knowledge and action in rural Bihar, India. Another standardized patient project is ongoing in Kenya.

With some parallels to the Fitzpatrick study above, Currie et al. 2011 reports on sending simulated patients to physicians in China with instructions to describe flu-like symptoms. Some patients were randomly assigned to also say, “I learned from the internet that simple flu/cold patients should not take antibiotics.” (I’m not sure how they pronounced the “/”.) The result was fewer antibiotic prescriptions, fewer drug expenditures, more information on drug side effects, and – here’s the clincher – ruder doctors. (So I guess you have to choose: Friendly doctor or appropriate prescription. Can’t we have both?)

Mystery shoppers were sent both to general shops and to drug shops in Tanzania to test treatment and referral policies in Hetzel et al. 2008. General shops are the first place that 25-30% of people go with a fever, and their treatments were substandard.

3. Testing for corruption

Dizon-Ross et al. 2014 sent mystery clients to test for mistargeting and corruption in programs distributing free bed nets to pregnant women in Ghana, Kenya, and Uganda. Male mystery clients — who were visibly not eligible — were sent to distribution centers to try and obtain a subsidized bed net (by any means necessary!). Across the three countries, fewer than 5% of mystery client attempts were successful in getting a net.

4. What about outside of health?

Most notably, a brief search turned up little outside of the health sector: This seems like a missed opportunity (although mystery shoppers were used to test the competency of librarians in Kenya, so that’s something).

One article mentions the use of mystery shoppers to monitor the performance of workers in one-stop government centers in Georgia, but there is no research reported. (One-stop government centers seek to make legal processes easier by offering many government services in one place.) The growing field of impact evaluations in governance (see here and here, for example) might be a natural space to use this tool, testing the speed and quality of service provision.

Where else are researchers already using mystery shoppers in development research, or where else should they be?

Update: A couple of readers immediately pointed me to examples of exactly what I proposed above!

Banerjee et al. evaluate four potential interventions to improve police performance in Rajasthan, India. They then use decoy visits to the police station to test. “Interestingly, the visits of decoy surveyors pretending to register crimes at police stations, which were initially conceived of primarily as a method of data collection, also had an effect of their own on police behavior–suggesting that a simple increase in monitoring may have a positive effect on police performance.”
Bertrand et al. (2007) doesn’t use the phrase “mystery clients” or “decoy visits,” but they offer a bonus to individuals to obtain a driver’s license in India and find that the individuals are likely to get a license but not to pass the driver’s test and yes to pay a bribe.
Gine et al. (2014) use mystery clients to evaluate services as financial institutions in Mexico. Researchers varied the financial sophistication and appearance of the client, among other variables. “Staff voluntarily provides little information about avoidable fees, especially to auditors trained to reveal little knowledge about the market. In addition, clients are almost never offered the cheapest product, most likely because staff is incentivized to offer more expensive products that are thus more profitable to the institution.”

This article was first published by the World Bank’s Development Impact blog. Publication does not imply endorsement of views by the World Economic Forum.

To keep up with the Agenda subscribe to our weekly newsletter.

Author: David is a Senior Economist in the Chief Economist’s Office for the Africa Region of the World Bank.

Image: Charli Crosby, 5, points to a doll in the window of an American Girl store at The Grove mall in Los Angeles November 26, 2013. REUTERS/Lucy.

Don't miss any update on this topic

Create a free account and access your personalized content collection with our latest publications and analyses.

License and Republishing

World Economic Forum articles may be republished in accordance with the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International Public License, and in accordance with our Terms of Use.

The views expressed in this article are those of the author alone and not the World Economic Forum.