Syndicate content

development impact links

Weekly links January 18: an example of the problem of ex-post power calcs, new tools for measuring behavior change, plan your surveys better, and more...

David McKenzie's picture
  • The Science of Behavior Change Repository offers a repository of measures of stress, personality, self-regulation, time preferences, etc. – with instruments for both children and adults, and information on how long the questions take to administer and where they have been validated.
  • Andrew Gelman on post-hoc power calculations – “my problem is that their recommended calculations will give wrong answers because they are based on extremely noisy estimates of effect size... Suppose you have 200 patients: 100 treated and 100 control, and post-operative survival is 94 for the treated group and 90 for the controls. Then the raw estimated treatment effect is 0.04 with standard error sqrt(0.94*0.06/100 + 0.90*0.10/100) = 0.04. The estimate is just one s.e. away from zero, hence not statistically significant. And the crudely estimated post-hoc power, using the normal distribution, is approximately 16% (the probability of observing an estimate at least 2 standard errors away from zero, conditional on the true parameter value being 1 standard error away from zero). But that’s a noisy, noisy estimate! Consider that effect sizes consistent with these data could be anywhere from -0.04 to +0.12 (roughly), hence absolute effect sizes could be roughly between 0 and 3 standard errors away from zero, corresponding to power being somewhere between 5% (if the true population effect size happened to be zero) and 97.5% (if the true effect size were three standard errors from zero).”
  • The World Bank’s data blog uses meta-data from hosting its survey solutions tool to ask how well people plan their surveys (and read the comments for good context in interpreting the data). Some key findings:
    • Surveys usually take longer than you think they will: 47% of users underestimated the amount of time they needed for the field work – and after requesting more server time, many then re-request this extension
    • Spend more time piloting questionnaires before launching: 80% of users revise their surveys at least once when surveying has started, and “a surprisingly high proportion of novice users made 10 or more revisions of their questionnaires during the fieldwork”
    • Another factoid of interest “An average nationally representative survey in developing countries costs about US$2M”
  • On the EDI Global blog, Nkolo, Mallet, and Terenzi draw on the experiences of EDI and the recent literature to discuss how to deal with surveys on sensitive topics.

Weekly links January 11: it’s not the experiment, it’s the policy; using evidence; clustering re-visited; and more...

David McKenzie's picture
  • “Experiments are not unpopular, unpopular policies are unpopular” – Mislavsky et al. on whether people object to companies running experiments. “Additionally, participants found experiments with deception (e.g., one shipping speed was promised, another was actually delivered), unequal outcomes (e.g., some participants get $5 for attending the gym, others get $10), and lack of consent, to be acceptable, as long as all conditions were themselves acceptable.” – caveat to note-  results are based on asking MTurk subjects (and one sample of university workers) whether they thought it was ok for companies to do this.
  • Doing power calculations via simulations in Stata – the Stata blog provides an introduction on how to do this.
  • Marc Bellemare has a post on how to use Pearl’s front-door criterion for identifying causal effects – he references this more comprehensive post by Alex Chino which provides some examples of its use in economics.

A few catch-up links

David McKenzie's picture
Our links are on break until the new year, but here are a couple of catch-up links now our job market series has finished:
  • BITSS had its annual conference (program and live video for the different talks posted online). Lots of discussion of the latest in transparency and open science. Includes a replication exercise with all AEJ applied papers: “69 of 162 eligible replication attempts successfuly replicated the article's analysis 42.6%.  A further 68 (42%) were at least partially successful.  A total of 98 out of 303 (32.3%) relied on confidential or proprietary data, and were thus not reproducible by this project.” And slides by Evers and Moore that should cause you to question any analysis done using Poissons or Negative Binomials.

Weekly links November 16: Remembering TN, targeting vs universal transfers debates, farcical robustness checks, bad replication techniques, and more...

David McKenzie's picture

Weekly links November 9: a doppelganger U.K., conditional distributions of journal decision times, invisible infrastructure, and more...

David McKenzie's picture
  • The Wall Street Journal discusses the synthetic control method as a way to understand Brexit (gated): “There are small differences in the various studies, but they all use Prof. Abadie’s method as the basis for constructing a “doppelganger” U.K. from other similar advanced economies, such as the U.S., Canada, France and the Netherlands. They reach similar conclusions, suggesting the British economy at the start of 2018 was around 2% smaller than it would have been had the 2016 referendum gone the other way”
  • Market-level experimentation: In the Harvard Business Review, How Uber used synthetic control methods combined with experiments to decide whether to launch Express Pool.

Weekly links November 2: harnessing shame, measuring markets, African safety nets and apprenticeships, rugby, and more...

David McKenzie's picture
  • “The average number of new social safety net programs launched each year in African countries since 2010 exceeded 10” – Kathleen Beegle on the Africa Can End Poverty blog discusses the rise of social safety nets in Africa.
  • The Declare Design team remind you to stratify your cluster-randomized experiments by cluster size.
  • With the job market coming up, a paper on the characteristics of “job market stars” – one factoid is that in development more than half the stars are female, compared to only 20% of all stars...another is that “not a single star student for six years running has taken a permanent job in industry”.
  • On VoxDev, Gordon Hanson and Amit Khandelwal discuss using night-light intensity to measure markets- with a comparison to what daytime satellite imagery reveals, and a note that combining the two provides the best results – “daytime imagery is particularly well-suited for defining the extent of market areas, and that nightlight imagery is useful for capturing the intensity of activity within these market boundaries”

Weekly links October 19: Men are from Mars, Women are from Venus – but only if you live in a rich and equal country, updates in randomization inference, graduation programs vs cash, small clusters not such a problem?

David McKenzie's picture

Weekly links October 12: should you decide on ethics by polling, beware the uneven treatment probabilities, roads are good, and more...

David McKenzie's picture
  • Stephanie Schwartz asks “are research ethics a question of public opinion?” on the Political Violence at a glance blog -  which discusses a new study that asked both research subjects and scholars their opinions on the acceptability of different research designs. Interesting discussion, particularly around what to do when the two differ – e.g. “A human rights advocate wants their interview with a researcher to be on the record. But the researcher worries that disclosing the subject’s name might put them in harm’s way. Does the researcher follow the participant’s understanding of “acceptable risk” and publish their name? Or do they follow their own instincts and keep the source anonymous?” (h/t This Week in Africa).
  • On the Future Development blog, Ariel Benyishay and co-authors discuss how they used satellite data to evaluate a USAID rural roads project in Palestine – using a diff-in-diff approach they compare nightlight in 750 meter grid cells shortly before, during, and shortly after the roads were rehabilitated to those in not-yet-improved cells. The report has some discussion of the many challenges involved, such as how to interpret an increase in nightlights, dealing with cells which have multiple roads treated, and the problem of potential spatial reallocation of economic activity.

Weekly links October 5: a new vision for social sciences in Science, doing development at non-R1, advice on jobs and on the media, and more...

David McKenzie's picture
  • Vision statement from Tage Rai, the new social and behavioral sciences editor for Science on what they are looking for: “I feel that our strength is the ability to bridge across social sciences in a way that very few outlets can and at a level that none can match. Therefore, we will be emphasizing papers that cross over major fields more strongly than ever before (e.g. psychology and anthropology, economics and political science, sociology and computer science).... The other major concern that I encounter has to do with scientific practice and reproducibility concerns within the social sciences. Science is actively engaged with these issues and continues to consider best practices going forward. ... First, only a small percentage of papers are sent out for in-depth review. For these papers only, authors will be asked to upload their data to an online repository accessible to reviewers.... I will host a twitter Q&A to answer any questions people may have about publishing with Science. For example, I am often asked about formatting for Science. As most papers are rejected, my approach is to be pretty loose about formatting, with the understanding that if a paper moves further into the process, we can revisit questions of word length and formatting at that time.”
  • Following up on my posts (part 1, part 2) this week on doing development in liberal arts colleges, Shreyasee Das has a thread on the challenges faced doing development research at non-R1/non-LAC schools.
  • Job market advice from Marc Bellemare, especially for those doing agricultural economics.

Weekly links September 28: the peril of meetings, endogenous responses mess up big data uses, what 600+ development papers tells you about our field, and more...

David McKenzie's picture