Syndicate content

Recent comments

  • Reply to: Using Case Studies to Explore and Explain Complex Interventions   2 days 19 hours ago

    Thanks for a nice post, Michael, expanding on your previous paper on external validity and hipply updating the example of a complicated intervention from a wristwatch to a smart phone.

    I continue to think that 'complex' isn't such a helpful construct but your overall point (learn more about why & how stuff works in particular places by exploring causes of effects) and your example is useful in making the point tangible.

    I'd like to further push a point that you & Rick (in comments) raise because it just doesn't seem to be sinking in for some. Case studies are almost always better when purposively selected. Gerring, Lieberman, QuinnPatton & several others have written usefully on this but I'll reiterate a bit.

    The following happens too much (as a caricature): 'we didn't understand our impact evaluation results, so we randomly selected some villages to ask a focus group composed of randomly selected people to ask "what gives?"'

    Kudos to people doing deeper dives (less kudos to over-reliance on focus groups) but pick cases that will really support more comprehensive learning about how a program & the world interact.

    To follow Michael's advice, undo this idea: random = better. And don't replace it with: convenience = fine.

    Where and with whom can you learn about the 'median impact narrative' (to borrow from Wydick)? Good, go there. Where are there high scores on the dependent variable? Super, plot the route to those villages/forms/countries. Low performers? Yup, seek them out intentionally, even if they are harder to find. A similar approach can be taken for variation on theoretically meaningful (meaning playing a hypothesized-to-be-important a role in the theory of change) observable independent vars, to inform deeper dives amongst those likely to have instructive variation on important unobserved (oops!) or unobservable variables. As you note, variations in implementation progress or quality can also be useful in pinpointing where small n (quant &/or qual) deep dives can facilitate more learning -- & monitoring data can ideally guide selection to look at heterogeneous cases.

  • Reply to: Using Case Studies to Explore and Explain Complex Interventions   5 days 20 hours ago

    Thanks Rick, and Fiona. Yes, indeed, to both comments. "Off diagonal" cases -- or "negative" as well as "positive" deviance -- is needed to get a better sense of the process mechanisms at work. We tried to do that in the MENA study I cite, though we could only use secondary sources (since, as you might imagine, it's hard to get officials to let anyone see their worst schools and clinics, etc.). Either way, the more general principle is asking 'Of what is this a case?' to any proclaimed instance of a phenomena. The answer requires knowledge of the nature and extent of the broader distribution.

  • Reply to: Using Case Studies to Explore and Explain Complex Interventions   1 week 2 days ago

    I agree with Rick's comment that you also need to also look at places where things don't work well. Otherwise you may find that what you think is the crucial path in the route to success could also be a route to failure. In other words you need to compare causal mechanisms for both successful and unsuccessful projects.

  • Reply to: Using Case Studies to Explore and Explain Complex Interventions   1 week 3 days ago

    Re "Conducting case studies in some of these exceptional places – in Palestine (education) and Jordan (health) – helped unpack the causal mechanisms by which, in these contexts, extant policies were being transformed into superior outcome"

    To me this quote highlights the need to put more cross-case analytic work into studying the many situations where things _don't_work_ and then identify the False Positives, i.e. the "positive deviants" which can then be the focus of case studies to identify the causal mechanisms at work that might have the potential to be replicated. Speaking more generally, case studies should emerge/follow on from case selection strategies.

  • Reply to: Conditional on your parents, does your country matter for early childhood human capital? Surprisingly no!   1 week 6 days ago

    First, thanks David for writing this up and offering his reactions.

    I too am curious whether the finding would generalize to countries with stronger government intervention in early childhood. (I'd love to hear of any samples that had sufficiently many child refugees/immigrants to investigate this.) Regardless, I find it interesting that the many other possible advantages of income and country seem to matter so little.

    I should also clarify in response to Abhijeet's comment that the paper's main findings and conclusions are only about early childhood human capital (hopefully this is clear in the paper). It has very little to say about the mechanisms for human capital formation or the possible divergence between countries after school starts, although I have investigated those topics elsewhere.