Syndicate content

The Danger of Stories

David Evans's picture

Who doesn’t love a great story? In development, we have wonderful stories to tell. We tell stories about children going to school because of cash transfers in Tanzania and about women in Costa Rica who are empowered through their jewelry businesses. The World Bank recently sponsored a day of development storytelling. Armendáriz de Aghion and Morduch recount the following story of a microcredit borrower from Mexico:
 

“Consider the story of Mrs. Braulia Parra, who lives with a family of seven in a poor neighborhood in Monterrey, Mexico, in a home with cardboard walls and dirt floors. Illiterate and inexperienced in the workplace, Mrs. Parra took her first $150 loan from ADMIC, a local microlender. The loan allowed her to buy yarn and other sewing supplies to make handsewn decorations. Each week she sells about one hundred handmade baskets, dolls and mirrors, going door-to-door in her neighborhood. After ten loans, Mrs. Parra had earned enough to install a toilet in her modest home, as well as an outdoor shower. Building a second floor was next in her sights.”

These stories inspire us, but we have to be careful. Stories are dangerous.
 


Stories are dangerous because they can be wrong. One program I am familiar with provided cash transfers to poor households. In the pilot, the government rolled out the program to a set of randomly selected treatment villages and compared them to a group of control villages. The program had myriad positive impacts (health, education, and much more!), but one area where the program did not have a clear impact was in housing materials. Yet when the researchers visited beneficiary villages and spoke to participants, they highlighted how they had improved their houses with the resources. The data showed that housing did indeed improve in participating villages: It just didn’t improve any more there than in comparison villages. In other words, the program was not responsible for housing improvements. But people received resources and they improved their houses, so their story (when speaking to the donors, at least) was that these resources had led to home improvements. If my goal were to start a program to improve housing, and I focused on the stories I heard, I might say: This is a great way to improve housing! I would be mistaken. Rather, it’s a great way to let households invest in whatever they most need, which turns out to be insurance and livestock rather than housing.

In the United States, moving anecdotes in which parents observe their children develop the symptoms of autism soon after vaccination led many people not to vaccinate their children long after the vaccination-autism link had been disproved. In both of these examples, through no ill will of the storytellers, there is potential for grave policy errors.

Stories are dangerous because they can be too simple. The economist Tyler Cowen gave a great talk (video; transcript) on stories where he explained that “narratives tend to be too simple. The point of a narrative is to strip it way … Most narratives you could present in a sentence or two.”
 

Tyler Cowen

Here are three of these kinds of stories: We lent this woman 20¢ to start her own bamboo stool business and she lifted herself out of poverty. We gave this child’s family $3 per month and they were able to send her to school. We provided training and women now have their own businesses and more freedom.

What do these stories miss? They miss that reality is messy and complicated. These stories miss, perhaps, that only a small fraction of beneficiaries experienced these benefits, or that the program only worked in certain contexts, or that not all handsewn decoration producers are equally effective entrepreneurs.

The inspiring microcredit story above and others like it have played a major role in motivating the microfinance movement, whereas evidence from randomized trials in India and the Philippines tell a much more complex tale, where microcredit has some positive impacts (lower spending on alcohol and tobacco in India, better ability to cope with risk in the Philippines) but also some … non-positive impacts (no increase in consumption in India, slightly lower subjective well-being in the Philippines).

The Upside. At the same time, stories can flesh out what quantitative narratives miss. In an evaluation of conditional cash transfers in Tanzania, we observed that beneficiary households were purchasing more livestock, especially chickens. Only through stories told in focus groups did we understand that these were seen as small business opportunities (selling eggs and chicks).

Take Away. The philosopher Bertrand Russell wrote that one goal of education is to enlarge people’s sympathy:
 

"It may only go so far as sympathy with suffering which is portrayed vividly and touchingly, as in a good novel; it may, on the other hand, go so far as to enable a man to be moved emotionally by statistics. This capacity for abstract sympathy is as rare as it is important."

Most of us aren’t quite there yet: Vivid and touching tales move us more than statistics. So let’s listen to some stories. Let’s be moved. Then let’s look at some hard data and rigorous analysis before we make any big decisions.

Photo Credits:
1) Pratham Books
2) Yuri Kozyrev (2002)/WB

Follow PublicSphereWB on Twitter
 

Comments

Submitted by anonymous on

This is a great point. We need better data, but the stories are a good way to make people more receptive to the conclusions of your data, through empathy, as you say. Thanks for sharing.

Submitted by Anonymous on

You are too kind to stories - and TED is a good example of that. At the start TED was a place where people came with insights they gathered based on data and turned them into a story. That's risky even in the best of circumstances as scientific consensus tends to change based on new evidence becoming available and retrofitting stories to reflect that is difficult.

However in TED as in life, stories soon became a way of doing away with the need for well-considered evidence and in recent years TED contributions are driven by form, not content. This produced a class of people who produce narrative for the sake of narrative. In our brave new post-modern world of false equivalency everything is an opinion so that it has become pretty much impossible to separate the woo from actual knowledge.

Submitted by Dave Evans on

I agree with much of what you say. Perhaps the challenge for those of us who work with statistical analysis is to come up with more compelling and creative ways to present data so that they can complete emotionally (as well as intellectually) with stories.

Thanks for this thoughtful post, Heather. I certainly agree with your call for more thorough thinking through how to rigorously collect AND ANALYZE qualitative data, and - having organized a lot of original quantitative data collection - I'm not at all blind to the problems with quantitative data. And I certain don't privilege all quantitative analysis: I do mostly RCTs, in which I have significant confidence.

Perhaps one of the challenges is that few researchers (that I know) can effectively speak both the language of quantitative analysis and of qualitative analysis, and so there can be a tendency for each group to dismiss the other. Alternatively, the qualitative research is thrown in to complement the quantitative analysis. (I would say more than dressing up and humanizing: Also to interpret quantitative findings.) Perhaps a more productive, constructive model, would be a research team where you have a PI with expertise in quantitative analysis and a PI with expertise in qualitative analysis. As I look on some of my own recent mixed method experience, it's clear that if we have invested the same resources into our qualitative work as we did in the quantitative work, we may have gotten something much more substantive. But I'm - by training - a quantitative researcher.

I do feel like most of the issues that we have with quantitative data (e.g., people potentially lying, external validity) are just as likely in qualitative data, although I'm happy to be proven incorrect. Unfortunately the Lincoln & Guba reference you link to is Harvard-gated; it looks like a great contribution to the literature.

Submitted by Suvojit on

Ignoring a good story is dangerous: We can qualify what constitutes a 'good story' - but insisting that stories be ignored and only statistics be looked at is quite extreme. As you say:

"What do these stories miss? They miss that reality is messy and complicated. These stories miss, perhaps, that only a small fraction of beneficiaries experienced these benefits, or that the program only worked in certain contexts, or that not all handsewn decoration producers are equally effective entrepreneurs"

But this is just bad story-telling. This is the story-telling of a politician or an ideologue and not that of a rigorous researcher. And to be honest, these stories probably do as badly as talking only about statistical average treatment effects.

Any researcher who dismisses stories is being too harsh and if stories = qualitative data, being this dismissive about them just demonstrates a lack of rigour. So let's not make decisions on stories alone. Let's also please not make decisions on statistics alone

Thank you for your thought. I agree that these simple narratives are bad qualitative research; this post isn't intended as a critique of qualitative research, even though I draw a couple of examples from focus group work. I DO have a critique of the simple stories that are often used to characterize the effectiveness (or ineffectiveness) of development projects (or vaccinations). In fact, I would not characterize high-quality qualitative work as "stories" at all: Good qualitative work pulls out common themes across the experiences of many people. But development work is rife with simple narratives that people use to motivate scale-up or to claim success.

But let me reframe my plea: Let’s listen to some stories. Let's be moved. Then let’s look at some hard data and rigorous analysis (both quantitative AND qualitative) before we make any big decisions.

Add new comment