Syndicate content

Recent comments

  • Reply to: Can temporary subsidies and agricultural extension build sustainable adoption?   1 week 4 hours ago

    I am intrigued by the first line. Developed countries continue to subsidize their agriculture: Farm subsidy in the USA and Common Agricultural Policy in EU. The Queen of England receives farm subsidy and there is a map of Manhattan that shows people who receive farm subsidy.

    BRAC reputation is an important factor here. I don't think we talk enough about the role played by the implementation agencies and how this should be taken into account in extrapolating external validity of RCTs

  • Reply to: Trouble with pre-analysis plans? Try these three weird tricks.   1 week 1 day ago
    Dear David,

    Thanks for that comment!  In fact, I had thought of linking to Registered Reports, but cut it in the interest of brevity.  I'm glad you have linked it here.

    Since you are here, could you also comment on two things?

    1) How does your initiative grapple with the asymmetry that, for some research questions, a significant finding is of greater importance than an insignificant finding (even a precise zero)?  Journals with limited space (and busy readers with limited attention) might need to condition on the actual findings, not just the analytical plan?  In other words, how many headlines begin with the phrase, “researchers FAIL TO discover new…” ?  It seems to me that the answer could lie in whether a journal’s commitment, based on the pre-analysis plan, precludes submission to other outlets.  (In economics, the norm is that a manuscript cannot simultaneously be under consideration at multiple journals.) If, for these journals, there was a multilateral agreement that this was allowed, you could get a pre-commitment that this journal will publish your study based on the plan, and then you are allowed to submit to top-tier outlets if you think you have a shot at one – i.e. if the finding is exciting, overturning priors, etc.  If it doesn’t pan out, this journal will publish whatever short report you want to submit with minimal editorial cost.  Is this the direction to be headed?

    2) How do we think about imprecise zeros?  A lot of the file drawer problem may be that the zero is imprecise: the survey instrument ended up very noisy, sample size ended up smaller than anticipated, attrition is high, compliance is low, the partner organization didn’t implement well, etc.  A journal editor might think that either a precise zero or a rejected null is interesting, but a poorly-implemented program or a data collection disaster isn’t.  (And isn’t that informative for meta-analyses either.)

    Best,
    Owen


     
  • Reply to: Trouble with pre-analysis plans? Try these three weird tricks.   1 week 1 day ago

    The author and I have been discussing his post via email, and I wanted to include one statement about a common question in regards to Registered Reports: not all null results are of interest to the wider research community, how will RRs not just be a dumping ground of uninteresting research? (not a direct quote)

    The answer to that question is that the editorial and peer review of Registered Reports can address that question before results are known. So, that review can assess if the proposed research question addresses a question that is of sufficient importance to the research community, such that a null result would be informative.

    Not all research questions meet this test. "I tried this thing that was not likely to have an effect, and lo and behold, nothing happened..." would not be a question that would likely be accepted before results are known.

    For many other research questions, however, true nulls are both important and difficult to disseminate (because nulls are easy to pick apart post-hoc). A question such as "Intervention X worked here, but others have had difficulty with it, how can we settle this dispute?" is a question whose results are informative regardless of outcome. By specifying the conditions ahead of knowing the results, strong biases are removed from the process.

    Obviously, work that is done poorly should not be published. Initial review will almost always look for various quality checks (manipulation checks, positive controls, etc), and those can be used to reject a paper after seeing their results, but only based on criteria that were established before the study was conducted.

    See more, especially the FAQ, on this page: https://cos.io/rr

  • Reply to: Trouble with pre-analysis plans? Try these three weird tricks.   1 week 1 day ago

    In regards to your final thought "Finally, if you are still having a hard time writing your pre-analysis plan, or you worry that your pre-analysis plans won’t pan out..", Registered Reports are a way to address both points (https://cos.io/rr). In this publishing format, peer review occurs once the pre-analysis plan is written, and the decision to publish is made before results are known. That way, the focus of that peer review process is on the theoretical interest of the research questions and of the ability of the proposed methods to address those questions. About 62 journals currently accept RRs. If your preferred journal does not, then go ahead and ask them to (https://osf.io/3wct2/). Finally, you can submit your pre-analysis plan (even if it is not part of one of those Registered Reports) to the Preregistration Challenge (https://cos.io/prereg) for a chance at a $1000 prize for publishing the results of preregistered work.

  • Reply to: What does a game-theoretic model with belief-dependent preferences teach us about how to randomize?   1 week 2 days ago

    This is a really helpful way of thinking through these issues. Another reason that public lotteries may not be desirable is that some interventions may not work well with this method. For example, when randomizing teachers into a treatment that involves promotion tournaments, teachers knowing who they are and are not competing with may or may not be desirable (depending on the promotion rule and the beliefs of the teacher about their promotion probability).