Do the demands that your research be “reproducible” overwhelm you? Have a solid idea of what “research transparency” means? Recently disagreed with a colleague about whether a dataset could be posted publicly or not? Whether you’ve experienced any of these symptoms or not, we invite you to join us Tuesday, September 10 at the World Bank for a full-day research symposium. The goal of this event is to try to ask and answer all of these questions, featuring open discussions with leaders in the field of reproducibility.
So much scrutiny, too little help?
The empirical revolution in development economics research has led to increased scrutiny of the reliability of applied research. Many development research projects rely on primary data to address specific questions. Hence it becomes urgent for applied development economists to document how the data generated in this process is collected, handled, and analyzed.
Now this is not particularly easy. Peer-reviewed journals are continuously searching for appropriate methods to ensure that published research satisfies the reproducibility condition. Yet, the most effective policies to achieve this goal have not been formally documented, and a wide range of policies are in place at different institutions. Similarly, donors and IRBs face a broad lack of guidance or consensus on what controls should be in place to ensure the quality of research outputs. This creates a bit of a cacophony for the well-meaning applied researcher: imagine you now have three grants per project, each with their own standards for reproducible research, which do not particularly nest each other … Scratch scratch.
Garden varieties of reproducible research standards feature both ex ante (or ”regulation”) and ex post (or “verification”) policies. Ex ante policies requires that the authors bear the burden of ensuring they provide some set of materials before publication and their quality meet some minimum standard. Ex post policies require that authors make certain materials available to the public, but their quality is not a direct condition for publication. Still others have suggested “guidance” policies that would offer checklists for which practices to adopt, such as reporting on whether and how various practices were implemented. The common thread through all clearly is—what material should be requested? In what format? How do I document all this?
Here to help
This upcoming September 10 workshop is a research-oriented symposium focusing on practical application of methods for transparency, reproducibility, and credibility in economics research. The event is collaboratively organized by DIME (World Bank), 3ie, BITSS (CEGA), IPA, and J-PAL. These organizations will lead discussions focused on their own successes and challenges in adapting research products to respond to these demands using tools like funding arrangements, IRB requirements, research public goods (some of them we blogged about here and here), and quality control in research and publication.
Then, researchers and practitioners will discuss how the transparency agenda impacts their work and how to improve expectations and practices as a discipline. These discussions will work towards a common understanding of research reproducibility; setting expectations for balancing open data and privacy; and moving towards practical steps for research to improve credibility and transparency without imposing undue burdens on researchers.
We hope to see you here at the World Bank next Tuesday! Space is limited to room capacity on a first-come, first-serve basis; the event will be broadcast via WebEx and recorded. World Bank staff should register through OLC and external attendees should RSVP here. Coffee and light lunch will be served.
Join the Conversation