Syndicate content

Recent comments

  • Reply to: Issues of data collection and measurement   10 hours 29 min ago
    Even the CONSORT guidelines, which mandate standardized reporting, does not get into that kind of detail. Journal templates with space limitations do not help, either. Finally, unless you're experimenting with these aspects, the bias is unknown. So, yes, agreed that these are important considerations but the solutions aren't easy to devise...
  • Reply to: Issues of data collection and measurement   11 hours 56 min ago

    Thanks for the quick response.

    In so far as these "Clever Hans effects" are a potential source of bias for many IEs that use their own data and in so far as that bias ordinarily goes in a direction that aligns with the preferences of research principals and the researchers, there is probably room for IE reports and papers to discuss how the potential for such biases has been mitigated. For instance, I don't think I've seen many reports / papers that discussed how the survey was introduced to respondents, what they enumerators knew of the treatment, and/or any differences between the contents of the survey instruments administered in the control and treatment groups. However, these details are probably as important as the many methodological details that are now de rigueur in the literature.

  • Reply to: Issues of data collection and measurement   12 hours 51 min ago
    Hi Ali,
    Thanks. Yes, (a) goes without saying. As for (b) you do the best you can but in many cases, especially long-term studies with multiple rounds of data collection, enumerators do learn...

    In the specific case where we have done this, it was in the fourth round of data collection with the results widely available in the media, so we calculated the downsides to be smaller than the upside of rewarding the people, most of whom had been repeat enumerators in our project over a period of five years...
  • Reply to: Issues of data collection and measurement   13 hours 41 min ago

    Treatment and Measurement - keep 'em separated.

    Berk, great post - measurement is a crucial area that needs much more interest and work in our field (development economics). Just wanted to say something that, I think, is understood (perhaps painfully obvious even) and you kind of get at it but just to be clear - it's important to keep treatment and measurement as separate from each other as possible. Here's what I mean. You say "My own view is that sharing impact evaluation results (or design at first follow-up) at the beginning of training at each round is greatly motivating for field teams". This is fine as long as: (a) field teams are not the same guys who administered the treatment (duh); and (b) field teams are kept away from knowledge of respondent treatment status (as much as this is feasible). So, what you say is fine as long as the surveyors are not clued in about the treatment status of their set of respondents (e.g., in what you say, it is fine to provide coarse results or sample averages or some such at first follow up but nothing more specific than that).

  • Reply to: Issues of data collection and measurement   14 hours 34 min ago
    Hi Andrew,

    Thanks. These are tough issues, on which the one-day workshop did not spend a lot of time on discussing. Generally, enumerators are blinded to treatment status, but, as you say they can find out. Economics treatments obviously are not, and not much we can do about that. Your point about other data sources is valid, but some IE-specific data collection will almost always be needed. Thinking hard about these issues and making calculated trade-offs, and triangulating information as much as possible, seems like the best I can think off...