A recent article in the New York Times describes a “stealth survey” to measure the difficulties in accessing timely health care. This U.S. government sponsored survey involves a team of “mystery shoppers” to pose as potential patients on the phone in order to measure the efforts required to schedule a doctor’s appointment as a new patient. The article assesses the reaction of four practicing doctors and finds a unanimous outcry among them: “This is not a way to build trust in government. Why should I trust somebody who does not correctly identify himself?” “If federal officials are worried about access to care, they could help us. They don’t have to spy on us.” And so on.
Mystery shopping is a standard tool in the private sector to measure quality of service. In fact there are trade associations such as the Mystery Shopping Providers Association that train and certify mystery shoppers (this would surely be a popular occupation if only it were better known!). The particular doctors quoted above may not be aware but the “mystery shopper”, or “mystery patient” in the case of health care, has been a long standing method assessing providers at all levels of the health system.
It is very difficult to measure a complex and multidimensional construct such as “quality of care” and the mystery patient is one key approach in a battery of tools that also includes direct observation of care, patient exit-interviews, and provider knowledge vignettes. Readers of this blog know that we are very interested in measurement themes and in the future I expect to return to the unique challenges of measuring quality of care.
This blog entry, however, focuses on another key aspect of impact evaluations that we will repeatedly return to: ethics. What disturbs the doctors quoted above (if we take their statements at face-value) is the dishonesty of the mystery patient method. Indeed the dishonesty of the method appears at first glance to fly in the face of one foundation of the ethical research principles laid out in the seminal Belmont Report of 1979.
The first basic ethical principle stated in the Belmont Report is “Respect for Persons”. This entails that individuals should be treated as autonomous agents. In practice, the respect for persons requires that subjects be given the opportunity to choose what shall or shall not happen to them – this principle forms the basis of informed consent. The deception inherent in the mystery patient approach appears to violate the autonomy of the study subject.
However the Belmont Report’s second basic ethical principle is known as “Beneficence”. This general principle has been codified by two complementary rules: (a) do no harm, but if the merits of the study justify some risk of harm or discomfort then (b) maximize possible benefits and minimize possible harms. The “minimize harm” principle necessitates judgments that weigh, for example, the discomfort of taking blood samples in a study population against the possible benefits of the information obtained from the blood analysis.
For the case of mystery patients, the possible harms to providers from the deception need to be assessed against the benefits of the proposed study. Time and again, ethical review boards have permitted the deception of the method due to the deemed importance of the research question at hand and the unique capacity of the method to gather certain information. In fact the deception at the heart of mystery shopping has prompted ethical guidelines that specifically govern the use of this method and the behavior of the mystery agent.
For example the Mystery Shopping Providers Association has a Code of Professional Standards and Ethics (which includes the bizarrely circular principle to “Conduct mystery shopping services in an honest and ethical manner “). The American Medical Association recently examined the ethical dimensions of mystery patients and gave a resounding endorsement conditional on certain requirements. I liberally quote from their conclusions on page 250 of the linked document:
Physicians have an ethical responsibility to engage in activities that contribute to continual improvements in patient care. One method for promoting such quality improvement is through the use of secret shopper “patients” who have been appropriately trained to provide feedback about physician performance in the clinical setting. A sound secret shopper program should include the following elements:
(1) All relevant parties, especially those to whom secret shoppers will be making unannounced visits, should be notified that this mechanism is being implemented in their practice setting.
(2) The information collected by secret shoppers should be used only to identify areas of improvement and not as a basis for punitive actions. Third parties should not have access to information collected by secret shoppers that includes personally identifying data.
(3) Feedback from secret shopper “patients” should not be relied on as the sole source of data for evaluating clinical performance.
(4) The use of secret shopper “patients” should not be implemented in a manner that adversely affects access to medical care by legitimate patients. For example, the need for urgent care (such as in the emergency department setting) must always take precedence over secret shopper “patients”.
In the view of the AMA, the ethical responsibility to improve care trumps the possible harm of deception as long as the consent of study subjects is sought. Indeed in a standard mystery patient study practitioners are told that, if they agree to participate, a mystery patient will contact them or attend their clinic but not told who it will be or when it may occur.
For development-related research there are at times additional ethical challenges. In a remote health clinic, staff typically has great familiarity with the catchment population. If informed of a possible visit from a mystery patient, these staff may quickly identify the atypical outsider as the mystery patient thus threatening the validity of the collected information. In this case, the presumed benefit from the information must be sufficient to even forego the informed consent requirement.
When deemed appropriate by a suitable review board, it may be ethical to both lie to your study subjects and also fail to warn them about the upcoming lie.
Update: Hours after I wrote this post, I find that the U.S. government has postponed the data collection exercise mentioned in the first paragraph due to criticism received. I presume the complaining doctors are members of the American Medical Association. I also presume they are unaware of their association’s own stance on mystery patients and the ethical responsibility to improve care.
Join the Conversation