In a working paper on the new LSMS-ISA (integrated surveys for agriculture) website, Brian Dillon describes the experience using phones in a research project he was working on with Diego Shirima, Geofrey Mwemezi and Msafiri Msedi. (You can also find a published version of this paper here). This project was designed to measure expectations that cotton farmers hold over outcomes such as weather, pest intensity, and crop yields in rural Tanzania. So data needed to be collected pretty often. This paper has broader relevance as well – as David discusses in a recent paper, frequent observations on variables with low autocorrelation can be a really effective way to increase power (like when working with businesses profits for instance).
So in what follows, I group Dillon’s lessons into two parts – some steps in making this work, and some more general lessons.
Steps to making this work:
1. Make sure you have a signal where the respondents are. Seems obvious, but heck, my provider doesn’t really work at my house (don’t ask).
2. Are you going to give them the phones? Dillon & co. gave them phones on loan. I’ve been involved in a project where we gave the respondents phones (the SMS was the treatment) and I’ve also heard of cases of folks giving out just the SIM cards – obviously the level of cell phone ownership in the community of interest are going to be key. Tip: get your phones from a reputable local supplier – we got ours from a local provider where the price was too good to be true, and it was…but in many cases local will be better because a) they tend to be cheaper than the US and Europe and b) you are much more likely to get a model that is familiar to your survey population).
3. Train folks on how this is going to work. This includes training them in how to operate the phones in order to respond, but also in things like how to maximize battery life.
4. Figure out charging. There might not be electricity where the respondents live. Dillon & co. signed contracts with people who ran charging stations in each village. In the case I was involved in, we gave them money for charging to use at local charging stations (we had a much more dispersed population).
5. Figure out a reasonable interview schedule. Dillon & co. called each farmer once every three weeks for (on average) a 27 minute interview. To help respondents remember when the call was coming, they gave them a laminated schedule (given my experience with how goats like to eat paper, lamination is a good investment).
6. Make sure the line will stay active. As those of you who travel know, if you don’t use a SIM card or put money on it for a period of time, it goes inactive and is eventually reassigned. Dillon & co. dealt with this by sending credit over the line to folks after each interview.
7. You do have to go back at some point. Dillon & co. went back a few months in and swapped out broken phones and non-working SIM cards, as well as getting other forms of feedback.
So what were the general lessons?
First: on cost, this does pretty well. Dillon & co. averaged 7 bucks an interview (which was 27 minutes). Yes, a baseline was necessary. But if you are going to collect multiple follow ups, this is a hell of a lot cheaper than going out there each time.
Second: should this be done everywhere? As Dillon points out if part of your area has network coverage and others do not, using phones for follow up will introduce bias. So the best call here is to do this where there is at least some part of every village that is covered (i.e. there is a place that wouldn’t introduce selection bias in the amount of effort it takes to reach it).
Third: do I really have to provide the phones? It depends on your population. If they all have phones, you don’t have to. But if they all don’t, you do. Whether or not they are the respondents’ to keep is up to you.
Fourth: what about attrition? Dillon & co. had a staggeringly low attrition rate – they averaged 191 of 195 respondents per round…
Fifth: how often should you call? Think about how often your mom calls you and apply the lessons from that – if it’s a short, fairly trivial interview, more often might be ok. Longer, more work – do it less often.
Sixth: a big plus – supervision. Doing this kind of interview let’s you put the supervisor in the same place with all of the enumerators at the same place. And the supervisor can check in with the local leaders since they’re likely to have a cell phone too.
Seventh: another plus – data entry. It’s easy to do computer assisted interviews with this method – the enumerators can enter the data as they talk. And this brings you the pluses of computer assisted surveys – including the built in, instantaneous data checks.
Eighth: yet another plus – cut down on call back costs. We’re running a survey right now where the survey firm is working on the second round of call-back interviews and this can be expensive as enumerators have to schlep out, possibly to a village they are no longer based in. With a phone interview, just call again.
Ninth: are they lying? When a (good) enumerator is face to face with a respondent, she or he can tell if the respondent is lying. On the phone, this is harder. But, in cases where the interview topic is pretty confidential, the phone can be useful – for example when asking tough questions with only a yes/no answer.
Tenth: dude, I lost my phone. So Dillon & co. have an interesting discussion on how to deal with “lost” phones. The bottom line is walk the fine line between keeping folks in the survey and them thinking that you are the fountain of endless phones.
Any other folks out there with experiences they want to share?
Join the Conversation