In his latest Letter from America in the Royal Economic Society’s newsletter, Angus Deaton says “your wolf is interfering with my t-value” (the title refers in part to regulations on hunting wolves in the American West) and talks about excessive regulation with NIH grants, and his concerns with the move towards trial registries:
“The regulators are concerned, not only with financial ethics, but with statistical ethics, not only our values but our t-values. In order to prevent the selective reporting of the results of randomized controlled trials, all clinical trials must be pre-registered with the government, and medical journals will not publish results where this protocol has not been satisfied. The American Economic Association is under pressure to maintain (and enforce) a similar registry for economic experiments …Along similar lines, it is argued that we must somehow take into account not only all of the calculations that have been done, but those that people chose not to do. That such regulations might have an adverse effect on creativity is noted, but dismissed, given the absolute necessity of preserving the purity of t-values. Bias should be regulated by sticking to randomized controlled trials, and t-values regulated by banning free form data exploration.
It is hard to believe that the science is so fragile that it needs this sort of protection. Would the case for (or against) a stimulus be affected by a new revelation about Keynes’ financial portfolio? … One of the problems is surely the reliance on randomized controlled trials, whose results depend, not on scientific advance or on convincing demonstration of a new mechanism, but on the precise circumstances under which they were conducted; if the experiment was tainted in any way, the results are automatically suspect. If RCTs are the only way of doing science, then we need the bureaucrats to guard the sanctity of their t-values. Yet, one wonders how so much good science managed to get done in the past.”
I have heard concerns of this type expressed about the possible use of trial registries and pre-analysis plans, but this is one of the few instances of seeing something in print, so it is useful as a conversation starter. However, I think Angus worries too much and is battling a strawman.
The issue, as I see it, is one that I know I sometimes fall into the trap of worrying about too – I know that pre-analysis plans and trial registries have the potential to be useful tools (see my last post), with a number of advantages. However, I also think that exploration of data and back and forth between data and theory is likely to yield unexpected insights that were unlikely to be anticipated in advance. When reading a paper that has used a pre-analysis plan, I would therefore not want to completely disregard anything that is not pre-specified and registered. But the worry is that although I know I am rational like this, and the other people I know who are likely to referee papers are also similarly rational like this, there is a fear that someone out there is less sophisticated and will say anything that is not registered is worthless. But I don’t see anyone making this argument, and I don’t think we give our community enough credit if this is the fear that is holding us back.
Likewise, I think the claim that RCTs results depend not on scientific advance or on convincing demonstration of a new mechanism, but on the precise circumstances under which they were conducted is doubly misleading – first, because many RCTs are designed to test particular theories, demonstrate proof of concept of some particular mechanism, etc ; second, because the general points that identification of genuine impacts requires not exploring 1000 different patterns in the data and choosing the one which has a large t-value is not specific to RCTs at all – indeed 3ie is building a trial registry that will attempt to register non-experimental impact evaluations as well.
It is useful to have these discussions about how this tool will be used before it is launched, but ultimately, it is up to us as a profession to use it wisely. I look forward to revisiting this issue once we have some evidence of how it is actually used in practice.
Join the Conversation