The potential perils of blogging about ongoing experiments

|

This page in:

One of the comments we got last week was a desire to see more “behind-the-scenes” posts of the trials and tribulations of trying to run an impact evaluation. I am sure we will do more of these, but there are many times I have thought about doing so and baulked for one of the following reasons:

·         Google and Hawthorne effects: as I’m trying to do more experiments with SMEs, many of whom use the internet, I am afraid of them coming across a description of the experiment on the web and this possibly biasing their behavior, their answers to surveys, or even their willingness to continue to be interviewed.

·         Media and the word “experiment”: For most of the experiments and evaluations being done with interventions to help poor people, the risk of anything more than trivial numbers of them finding out about the evaluation through the web are slim to none (and yet the University of California human subjects committee still insisted we add an email address to the informed consent form!). My more major concern is that a journalist who either doesn’t understand what we are doing, or who is looking for a sensational story, will pick up on the word experiment and write a story with the headline “World Bank experimenting on our poor!” or something like this. I think this fear is more of a concern working at the World Bank than at a university, and more of an issue working on evaluations of government programs than working with NGOs, so I think it is something that is more of a potential issue for us than our friends in academia. I don’t think the chance of it happening is that large in any particular case, but the risk is certainly there, and such a story could lead to problems in either implementation or in follow-up surveys.

·         Keeping partners happy: Much of the behind the scenes lessons, stories, mini-disasters, and other interesting tidbits come from interactions with partners – be they the intervention implementers, the survey firm, or the policymaker behind the intervention. However, these are all people you need to keep on your side, especially when things are going badly – so sharing a story of the lessons from the latest calamity in the field while it is still fresh might make for an entertaining blog post that may also help others avoid the same mistake, but risks causing umbrage to partners, especially if they are the reason for the problem occurring. Not to mention the political angle in some cases – I’ve already experienced one case where I wrote about an experiment that went wrong due to the partner organization inviting the control group to come along to make up for non-attenders among the treatment group and got complaints that I was blaming a client for this. On the other hand, once all has been resolved and the intervention has been completed, there may be more willingness on the point of view of partners to discuss what in retrospect look like minor hiccups along the road, despite seeming at the time much more serious.

·         What is the value added of such a pre-emptive post? Given the issues above, I feel like there need to be a strong case that there is good value-added from posting while the intervention and surveys are ongoing. While we try and entertain on occasion, a good story per se doesn’t seem reason enough – so either there has to be a lot of potential value to others in sharing this experience, or a lot of potential value in getting feedback on what we have done to make it seem worth blogging about something while it is ongoing. So expect some of this on occasion from me, but less than I would do if I wasn’t worried about the things above?

Readers – do you know of any examples where blogging about (or posting descriptions on a webpage like the IPA, JPAL, or DIME project descriptions has impacted on the project itself? Am I worrying too much?

Authors

David McKenzie

Lead Economist, Development Research Group, World Bank

Berk
April 08, 2012

David,

I had many of these concerns written in my first draft last week, but decided to cut them once the post reached almost 2,000 words. I don't think you're worrying too much and I, more than once, decided against blogging re: an ongoing experiment exactly for one of those reasons you outlined above.

Berk.

Steve
April 09, 2012

David,
Like Berk, I think your concerns are definitely valid and that this is an area to be really risk adverse since you never know when something that seems trivial to an academic can be totally twisted by a journalist, policymaker or politician. Eight years on and I get the same looks from policymakers when I say 'lottery' instead of 'ballot' or even worse 'losers' instead of 'unsuccessful applicants' when talking about our Pacific work.

Steve

Patti
April 09, 2012

Hi David. You can always share lessons from ongoing and past work by using substitute names, or simply by not divulging the region of the country, or perhaps not even the country, no? Many key lessons from my qual work are not all that context-specific.

David McKenzie
April 09, 2012

Thanks Patti - indeed, this is what Markus did in his post on the evaluation roller-coaster (http://blogs.worldbank.org/impactevaluations/the-impact-evaluation-roll…)

However, I do think detail is important in discussing the trade-offs, and makes it more concrete - so there is in my mind at least a big tradeoff between removing enough context to make it not identifiable, and still maintaining enough to make clear what is going on. Moreover, using pseudonyms is unlikely to solve the problem of sensitivity of partners - if I say my project is really suffering because company X made this type of mistake, I would still be worried about company X getting upset about it while I am still trying to work with them - even though X is a pseudonym.

Nevertheless, there are certainly times where you can share lessons without providing all the details, and we do try and do that at times.

Finally, I just find it a lot less satisfying to read when it becomes more generic and less specific in terms of what the evaluation is, what country it is in, etc.