Inside the Black Box: Why do things work?

There are now a variety of well-known experimental and non-experimental methods that economists use to learn whether a given program works or not. However, our tools for learning why or why not something works are much more limited.

In a useful new paper, Kosuke Imai and co-authors carefully set out the additional assumptions needed for researchers to move from estimating causal effects to also identifying causal mechanisms. They give a couple of examples from the political science literature to illustrate their points. I’ll try here to summarize some key ideas through applying them to an economics example.

Consider Business Training, a popular policy aimed at improving the productivity and increasing the incomes of small business owners. I’ll blog about new results from several randomized experiments on business training in future posts. But for now, suppose we find that business training is actually increasing business profits. Then we would like to know why?

Some possibilities:

• Business Training → Better Business Practices (like accounting and management) → Higher profits
• Business Training→ More knowledge about how to apply for credit → More credit → Higher Capital stock → Higher Profits
• Business Training → More motivated entrepreneurs → Higher work effort →Higher profits

Even if we have a randomized experiment which randomizes which owners get trained and which don’t, further assumptions are needed to learn which mechanism is operating. Suppose we want to understand how much of the improvement in profits comes from better business practices (appropriately measured) rather than through other mechanisms.

Imai and co-authors call the key assumption sequential ignorability. In our context, this amounts to assuming that conditional on whether or not someone has been trained, and on other observable controls, their level of business practices is independent of profits. This is quite a strong assumption – which would be violated for example if those with better business practices were also more educated, and if education had affects business profits through channels other than business practices.

Is this too strong an assumption? Remember it is an assumption conditional on observables, so puts us squarely back in the world of observational studies – if we can control for enough stuff, perhaps it is not so bad.

The money quote from Imai et al (p. 12):

“It is worth recalling that, in general, research with observational data is built upon a strong assumption that conditional on covariates the treatment variable is ignorable. Despite this, much can be learned from observational data. In fact, many social science theories result from simple observations and many experimental studies confirm the results of observational studies”.

So under this sequential ignorability assumption, we can estimate:

Profits = a1+ b1*Business Training + c1’X + e1, and

Profits = a2+b2*Business Training + c2’X + d2*Business Practices + e2

And then get b1, the total average treatment effect of business training; b1-b2 as the estimate of the average causal mediation effect (ACME) of how business training affects profits through the channel of business practices, and b2 as the average direct effect, measuring how business training affects business profits through other channels. Imai et al also provide non-parametric ways to estimate this in cases where linearity is not desirable.

Fine you say, but I don’t believe this assumption, can’t I do better?

One intuitively appealing approach might be to try and randomize more steps along the causal chain. For example, if we could use our first experiment to estimate the causal effect of business training on business practices, and then do a second experiment where we somehow persuaded businesses to randomly alter their business practices (by sending in useless consultants for example) to identify the causal effect of business practices on profits, couldn’t we just multiply the two effects together?

The answer is that:

• If the treatment effects are constant, then this works
•  if the causal effects vary across individuals, this only works if the sequential ignorability assumption holds (see section 7.1 of the paper). Intuitively, the concern is that it could be that the type of people for whom business training leads to big changes in business practices may not be the type of people for whom changes in business practices make that much difference for profits.

So what else can we do? Imai and co-authors discuss several other approaches:

• Use sensitivity analysis to see how much of a violation of the assumption there would need to be to rule out your causal mechanism
• Randomized experiments where the mediating variable (aka our channel of interest) is randomly varied within each treatment status
• Examining treatment effect heterogeneity according to pre-treatment covariates – they discuss some pros and cons of this approach, and note it also requires additional assumptions.

There are a number of alternative approaches to exploring causal mechanisms, including qualitative approaches, but to date most papers in economics typically report average treatment effects, and then sometimes use theory to suggest where we might see heterogeneity in these effects.  I therefore believe learning what other approaches people are trying, especially when they clearly set out the assumptions needed for them to work, offers some encouraging avenues to take economic impact evaluations going forwards.