The new book Uncontrolled by Jim Manzi has attracted a lot of recent press (e.g. see Markus’ recent post for discussion of David Brooks’ take, or this piece in the Atlantic), and makes the argument that there should be a lot more randomized experiments of social programs. I was therefore very interested to order a copy and just finished reading it.
The book is one-third philosophy and history of establishing cause-and-effect in science, one-third discussion of how experiments have been used in business and social sciences to date, and one-third politics. I found it half-interesting, but also a missed opportunity to really learn from what has been done in business.
Jim Manzi is the founder of Applied Predictive Technologies, a company which has developed and licenses software for automated test-and-learn randomized experiments for businesses. The company’s website says it has worked with Starbucks, Victoria’s Secret, Lowes, Staples, McDonalds, Walmart, and many other well-known companies. The big missed opportunity in my opinion is that the book provides very little detail on what businesses have actually learned, achieved, or gained as a result of randomized experiments. We get the relatively well-known story of Capital One and the fact that it runs more than 60,000 experiments per year, get a nice anecdote that at Harrah’s casinos the three key rules for staff are “don’t harass women, don’t steal, and you’ve got to have a control group”, and that Google claims to have run 12,000 randomized experiments in 2009, with about 10 percent of these leading to business changes. But we get almost nothing on the experiments his company has been involved in, or the insights that businesses have learned from all these experiments.
This is what I consider the big missed opportunity. I suspect that part of the reason is that the results of these experiments are mostly proprietary, with companies reluctant to share results that might help their competitors. But APT’s website has at least a number of case studies, describing for example how a bank used repeated experimentation to figure out which branches to allocate specialized staff to, increasing annual profit by “millions of dollars”; how a specialized retailer uses these methods to measure how much online advertising in particular markets affected offline saleshttp://www.predictivetechnologies.com/en/index.cfm/solutions/retail/business-issues/internet-advertising/measuring-offline-sales-from-online/, etc. These case studies show that it is at least possible to share some of these examples, and provide more compelling evidence of the types of changes businesses have made in response to experimentation. Documenting some of the examples where monetary gains through sustained implementation of marginal changes suggested through experimentation would help better sell the case for these experiments – and provide better evidence of the book’s subtitle (The surprising payoff of trial-and-error for business, politics and society). Better still would be insights about human behavior that have been realized in response to these changes.
What we do get are results which are useful at a very high level, but are perhaps less surprising and novel:
· “There is no magic” – experimentation typically creates only marginal improvements - a failing company with a poor strategy cannot blindly experiment its way to success. Innovative ideas rarely work, and those that do typically create improvements that are small compared to the size of the strategic issues they are intended to address.
· The sine qua non is executive commitment – the CEO or president must legitimately desire reliable analytical knowledge of the business – he also gives examples from US policy where the Institute of Education Sciences had a director that forcefully advocated using randomized trials, or the National Institute of Justice had a director in the early 1980s who was a supporter or randomized trials.
· Most results are conditional – Manzi talks a lot about the external validity issue – in his words business and social science applications have “high causal density” – so replication and iterative testing to find under which conditions things work is important – which is achieved in part in business by lowering the cost of each experiment so that many replications can be done. An interesting statement (p. 204) is that:
“The experimental revolution is like a huge wave that has lost power as it moved uphill through topics of increasing complexity and holism. Physics was entirely transformed; therapeutic biology required statistical experimentation due to higher causal density but could often rely on the assumption of uniform biological response to reliably generalize findings from randomized trials; the yet-higher causal densities in social sciences make generalization from even properly randomized experiments hazardous, and the integrated complexity of certain topics in social science appears to be fully impervious to experimentation”. But on the other hand “recognizing the difficulty in generalizing an experiment should not blind us to is importance and power…without internal validity, … there is no point in worrying about external validity”
Chapters 11 (on business experiments) and 12 (on experimental social science) of the book are likely to be of most interest to our readers. The chapter on experiments in the social sciences overviews the use of experiments in the fields of crime (e.g. broken-windows policy); welfare (e.g. mandatory work requirements to try and get people off welfare and into jobs); education (e.g. school choice, pre-school curricula); economics (his claim is that experiments have focused on the periphery of the core economic functions of production, logistics and sales in the major sectors of the economy); political science (mostly get out the vote programs). His conclusion when reviewing these studies is that “programs that attempt to improve human behavior by raising skills or consciousness are even more likely to fail than those that change incentives and environment” – so e.g. creating choice for students in which schools they can go to, rather than experimenting with curricula and training within schools is more likely to lead to sustained gains in academic performance”. All of the experiments in developing countries and the lessons learned from them are reduced to part of a single sentence on page 195. I get that the book is aimed at a U.S. audience, but surely there are some insights from this work for policymakers in developed countries as well?
All in all then, this book is interesting reading, but not as good as I had hoped it could be. The lesson of trial-and-error with thousands of relatively low-cost experiments designed to make marginal improvements is one that could be useful in many government bureaucracies (and indeed in our own bureaucracy), but the assumption of Manzi that there should be a rational status quo bias where we start where we are and try to improve iteratively since where we are isn’t too bad and is the result of trial and error through social evolution may be less defensible or appealing in many developing countries – where the big challenge is not marginal changes to innovate around the global frontier, but rather massive changes required to move from a situation where many things work badly to one in which most work well.
Join the Conversation