The March 2011 issue of the Harvard Business Review has “a step-by-step guide to smart business experiments” by Eric Anderson and Duncan Simester, two marketing professors who have done a number of experiments with large firms in the U.S. Their bottom line message for businesses is:
“[Managers] need to embrace the “test and learn” approach: Take one action with one group of customers, take a different action (or often no action at all) with a control group, and then compare the results. The outcomes are simple to analyze, the data are easily interpreted, and causality is usually clear. The test-and-learn approach is also remarkably powerful. Feedback from even a handful of experiments can yield immediate and dramatic improvements in profits.”
They go on to give examples of some of the types of experiments firms are carrying out, such as Capital One testing the effectiveness of free balance transfers from other credit cards through a pilot experiment on treatment and control samples of prospective customers, direct-mail houses, catalog companies and online retailers testing discount strategies and how often to contact customers, and retail stores testing promotional strategies. These examples of how successful businesses use experiments may be helpful in persuading some types of clients in development projects of the value of experimentation.
They then offer 7 rules for running experiments within companies and discussion of the process of overcoming internal resistance within companies to experimentation. Some insights of this for thinking about development experiments:
- When key outcome data is being collected automatically, there is the opportunity to do hundreds of simple experiments, with the expectation that only a small number will prove profitable. In their context, banks and retailers can automatically observe whether customers are changing behavior or buying new products. There are a number of development contexts which fit this criterion, and where experimentation could therefore be the rule rather than the exception: Banks offering loans automatically observe repayment behavior, Schools testing students automatically measure test scores, anthropometrics may be measured as part of many health projects, worker productivity in some jobs may be easily observed by firm owners, time taken to issue various permits should be automatically measured by government authorities. In these cases, rather than just the occasional large-scale experiment, continual experimentation could be very valuable.
- Thinking outside the box: “Tesco [the U.K. supermarket chain] allows relatively junior analysts at its corporate headquarters to conduct experiments on small numbers of customers. These employees deliver something that the senior managers generally don’t: a steady stream of creative new ideas that are relevant to younger customers.” The same is happening to some extent in development with NGOs, where young creative researchers are trying out innovative new ways to get people to save, insure, stay in school, and other good things. It would be fantastic to see the same happen more with government and World Bank projects – note the important thing here is not just to pilot new ideas, but to rigorously test them through experimentation so that we know whether they work or not.
- “The internal obstacles to experimentation are often larger than the external barriers. In an organization with a culture of decision making by intuition, shifting to an experimentation culture requires a fundamental change in management outlook”. This statement appears true not only of companies, but of governments and development organizations – in many cases experimentation is logistically feasible and can be done in a way that overcomes external concerns, but there is a resistance to moving from a culture of “this is the way it has been done” and “we know this works”, to one of dynamic learning.
Join the Conversation