Recently both the American Economic Association and 3ie have launched Impact Evaluation Trial Registries. The basic idea in both cases is for researchers to register in advance the details of an evaluation they are planning on doing. This has a couple of main purposes:
So what’s the difference between them/why are there two?
The American Economic Association AEA RCT Registry (https://www.socialscienceregistry.org/) is for randomized experiments. My understanding is that it is intended to cover both field experiments and lab experiments, and it places no restrictions on the country. It has started with an approach centered around getting people to put in some very basic information (to help address point 1 above), while making optional the ability to add more detailed pre-analysis plans etc. (essentially leaving it up to researchers at the moment how far they go in addressing point 2). A basic registration at the moment just requires 16 mandatory fields such as country, start and end dates for the intervention and for measurement, a basic description of the intervention, what the key outcome variables are, what the unit of randomization is, the planned sample size, and documentation of any IRB approval. You then have the choice to upload further information like a pre-analysis plan, survey documentation, or power calculations. For most fields you can choose whether they are public or hidden. Hidden fields are not viewable to others (although they can click on a link to request permission from you to see them), and when the planned end date of your trial passes, the registry will automatically publish these fields. As soon as you click submit the public fields appear online and you automatically get issued a trial registry id. Here is a list of the studies currently registered.
Concurrently, 3ie (the International Initiative for Impact Evaluation) has set up the Registry for International Development Impact Evaluations (RIDIE) (http://www.ridie.org/). It is designed to be a prospective registry of attempts to measure the causal impacts of development programs in low and middle income countries. On one hand it is broader than the AEA registry, because it is designed to also allow registration of prospective impact evaluation designs that will rely on non-experimental methods such as regression discontinuity, matching, or difference-in-differences. On the other hand, it is not intended to cover evaluations in developed countries, nor lab experiments. Like the AEA registry, it has a relatively short list of required fields, along with a larger number of optional questions and the scope to upload documents providing more information. The website is more professional looking than the current bare-bones AEA registry, and has a lot more supporting information in terms of background documents, FAQs, etc. For most fields you have the choice of whether you want this field to be public or private, with the fields you choose to keep private automatically then made public once the date you specify as the end of your trial occurs. They manually check entries for completeness before they appear online, and in my case, asked for some changes before approving. In order to try and generate initial sign-ups, they are having a lottery among the first 100 registrations.
There is no need to register in both of them. I have registered one study in both since it had both an experimental and a non-experimental evaluation component, but mostly just to try the two out.
How time consuming is it?/When should I register?
While preparing a full pre-analysis plan can be time-consuming, the basic registration of mandatory fields on either site can be very fast- you can easily register the basics of a study in half an hour I would say. Both sites then allow you to update at any time (keeping a record of the process), so you can then come back and add supporting documentation at a later time if you want.
Ideally studies should be registered prospectively. However, since these are new and impact evaluations can take several years to complete in many cases, both sites are keen to also have people register ongoing evaluations. This helps in point 1), although the role of the registry in helping pre-define outcomes and specifications is less useful when people have already collected data.
My personal feeling is that this process is most useful to me once I have done the baseline and launched the intervention, and then am planning out the schedule and questions for follow-up surveys. Many of the questions about sample sizes and timelines are unknown to me, or highly changeable, in advance of these steps, and I find the process of writing a pre-analysis plan is really useful for designing a follow-up survey. However, I have also been told by colleagues who work in health that medical journals typically require trial registration before baseline has even been collected, so this is something to consider if you work in those fields.
How is this going to be used?
I think the profession is sensibly taking a cautious approach to allow norms to evolve about how these should be used. Although the AEA has set up a registry, currently its journals do not require registration, although my prediction is that referees and sometimes editors will consider whether or not a study is registered when evaluating studies (taking account of the substantial existing pipeline of studies though). This is particularly likely to matter when the main results seem to be in a domain that is not obvious as the main domain of study (e.g. when a schooling support study finds significant results on pregnancy, but not on schooling) or when the authors focus on a particular dimension of heterogeneity in treatment effects (in which case whether this has been pre-specified will be of interest). A number of funders also seem likely to require registration, so over time these databases should build.
One of my biggest fears has been whether this information gets misused while the evaluation is underway (related to my post on the potential perils of blogging on impact evaluations while they are in progress). This includes concerns about Hawthorne effects if people in a study learn about what our hypotheses are, as well as concerns about media or politicians misinterpreting what research is trying to do. Therefore I am very glad that both sites allow much of the details to be kept private in the interim, which should lessen this risk substantially.
Disclaimer: I was part of a Design Task Force that 3ie and RAND put together when designing RIDIE, and also was involved in beta-testing the AEA registry.
- It is intended to help provide a better record of which evaluations are being attempted, which can i) aid meta-analysis efforts; ii) help serve against publication bias whereby studies with nil results or that have to get abandoned part-way through often don’t get published; and iii) potentially aid researchers looking to see which other evaluations are going on that might be complementary.
- It is intended to help add transparency and protect against selective reporting of outcomes, by asking researchers to specify in advance the main outcomes they are interested in looking at, and the main hypotheses they will test. See my post on pre-analysis plans for more on this.
So what’s the difference between them/why are there two?
The American Economic Association AEA RCT Registry (https://www.socialscienceregistry.org/) is for randomized experiments. My understanding is that it is intended to cover both field experiments and lab experiments, and it places no restrictions on the country. It has started with an approach centered around getting people to put in some very basic information (to help address point 1 above), while making optional the ability to add more detailed pre-analysis plans etc. (essentially leaving it up to researchers at the moment how far they go in addressing point 2). A basic registration at the moment just requires 16 mandatory fields such as country, start and end dates for the intervention and for measurement, a basic description of the intervention, what the key outcome variables are, what the unit of randomization is, the planned sample size, and documentation of any IRB approval. You then have the choice to upload further information like a pre-analysis plan, survey documentation, or power calculations. For most fields you can choose whether they are public or hidden. Hidden fields are not viewable to others (although they can click on a link to request permission from you to see them), and when the planned end date of your trial passes, the registry will automatically publish these fields. As soon as you click submit the public fields appear online and you automatically get issued a trial registry id. Here is a list of the studies currently registered.
Concurrently, 3ie (the International Initiative for Impact Evaluation) has set up the Registry for International Development Impact Evaluations (RIDIE) (http://www.ridie.org/). It is designed to be a prospective registry of attempts to measure the causal impacts of development programs in low and middle income countries. On one hand it is broader than the AEA registry, because it is designed to also allow registration of prospective impact evaluation designs that will rely on non-experimental methods such as regression discontinuity, matching, or difference-in-differences. On the other hand, it is not intended to cover evaluations in developed countries, nor lab experiments. Like the AEA registry, it has a relatively short list of required fields, along with a larger number of optional questions and the scope to upload documents providing more information. The website is more professional looking than the current bare-bones AEA registry, and has a lot more supporting information in terms of background documents, FAQs, etc. For most fields you have the choice of whether you want this field to be public or private, with the fields you choose to keep private automatically then made public once the date you specify as the end of your trial occurs. They manually check entries for completeness before they appear online, and in my case, asked for some changes before approving. In order to try and generate initial sign-ups, they are having a lottery among the first 100 registrations.
There is no need to register in both of them. I have registered one study in both since it had both an experimental and a non-experimental evaluation component, but mostly just to try the two out.
How time consuming is it?/When should I register?
While preparing a full pre-analysis plan can be time-consuming, the basic registration of mandatory fields on either site can be very fast- you can easily register the basics of a study in half an hour I would say. Both sites then allow you to update at any time (keeping a record of the process), so you can then come back and add supporting documentation at a later time if you want.
Ideally studies should be registered prospectively. However, since these are new and impact evaluations can take several years to complete in many cases, both sites are keen to also have people register ongoing evaluations. This helps in point 1), although the role of the registry in helping pre-define outcomes and specifications is less useful when people have already collected data.
My personal feeling is that this process is most useful to me once I have done the baseline and launched the intervention, and then am planning out the schedule and questions for follow-up surveys. Many of the questions about sample sizes and timelines are unknown to me, or highly changeable, in advance of these steps, and I find the process of writing a pre-analysis plan is really useful for designing a follow-up survey. However, I have also been told by colleagues who work in health that medical journals typically require trial registration before baseline has even been collected, so this is something to consider if you work in those fields.
How is this going to be used?
I think the profession is sensibly taking a cautious approach to allow norms to evolve about how these should be used. Although the AEA has set up a registry, currently its journals do not require registration, although my prediction is that referees and sometimes editors will consider whether or not a study is registered when evaluating studies (taking account of the substantial existing pipeline of studies though). This is particularly likely to matter when the main results seem to be in a domain that is not obvious as the main domain of study (e.g. when a schooling support study finds significant results on pregnancy, but not on schooling) or when the authors focus on a particular dimension of heterogeneity in treatment effects (in which case whether this has been pre-specified will be of interest). A number of funders also seem likely to require registration, so over time these databases should build.
One of my biggest fears has been whether this information gets misused while the evaluation is underway (related to my post on the potential perils of blogging on impact evaluations while they are in progress). This includes concerns about Hawthorne effects if people in a study learn about what our hypotheses are, as well as concerns about media or politicians misinterpreting what research is trying to do. Therefore I am very glad that both sites allow much of the details to be kept private in the interim, which should lessen this risk substantially.
Disclaimer: I was part of a Design Task Force that 3ie and RAND put together when designing RIDIE, and also was involved in beta-testing the AEA registry.
Join the Conversation