David’s post yesterday on migration got me thinking about the general problem of finding folks when you go back for the second round (or higher) of a panel survey. An interesting and extremely useful paper by Firman Witoelar on the LSMS-ISA website gives some guidance on this.
First, Firman makes the point that we ought to do this more often. He points out that there is work out there by Duncan Thomas and co-authors, as well as Kathleen Beegle and others that shows the folks missing in round 2 are different – in unobservables, and in observables that you aren’t observing. Indeed, Beegle & co. find that consumption growth of movers is much higher than those who stayed. So, as Donald Rumsfeld put it: “because as we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns -- the ones we don't know we don't know.” The point now is that now we should be suspicious – the fact that folks aren’t around means that there is some unknown (how they are different) that we really ought to know. This is particularly germane for impact evaluation where it could be the case that the intervention (or the hope of getting it) entices certain types to stay, or, on the other hand, that the benefits of the intervention make it possible for certain types to leave (or some other unknown form of bias). Bottom line, you need a theory (and better still some evidence) of where the missing respondents went, and why.
Second, Firman points out that in follow-up surveys, there are actually two types of households to consider. First, “tracking targets” which are “individuals who when moved, should be tracked, and when found, should be interviewed.” Second are “interview targets”, which are “those who need to be interviewed.” As he points out a) you may not want to track everyone, and b) you may want to interview people who weren’t in the baseline (e.g. returned migrants and other new HH entrants).
So this raises the question, whom to track? In a world with no resource constraints, the answer might well be everyone. But there are time and cost implications, so Firman gives us a couple of things to think about:
- Start with why you are tracking. This will help get you situated on the cost/quantity tradeoff spectrum. And consider doing a pilot before the main survey goes out so you can see what the numbers might look like – or else check in on the attrition fairly early in the survey.
- Set boundaries – this depends on the structure of your survey, but possible types include administrative borders or a simple distance limit (the projects office just might query the receipt for airfare to Vanuatu)
- Think about maybe doing all the movers who fall in the survey catchment area and then do a sub-sample of the long distance ones (at least you will change the unknown unknowns to known unknowns, and heck, with a big enough sub-sample over a representative area, they might be known).
And then there is the question of how to do it. Firman hammers home the importance of planning – not only about whom to get, but a) setting up forms in round 1 that give you a wealth of potential informants (with multiple phone numbers given the SIM switching that goes in countries with competitive cell phone markets), b) thinking about the timing of it all – economically it can make sense to wait until the end of the survey (so you only have to go to Vanuatu once) but this can mess with seasonality, c) making sure there is dedicated supervision and really persistent enumerators, d) making sure there is good communication and real time information flows between different teams and the supervision (including good quality tracking forms), and e) you don’t mess up the household IDs (i.e. you can link them but the IDs are unique). There is also a discussion of different tools to use in helping track people – ranging from the idea of using records (including church records) to photographs (which you can tie into computer assisted surveys very nicely) to incentives for both respondents and the tracking enumerators.
Finally, Firman also raises a central ethical issue. How do you pitch the possibility of follow up in the first interview? It’s not totally obvious (particularly if you don’t know how many waves there will be). And, in addition, in the planning phase it is worth thinking about how to deal with people who may not want some folks to know where they are – which could be a concern with any survey going after sensitive issues (e.g. domestic violence). So this whole issue doesn’t make doing surveys any easier – but Firman’s paper provides some really helpful guidance to make it not so painful.
Join the Conversation