I am currently in Malawi rolling out a firm survey with my colleagues Francisco Campos and Manuela Bucciarelli. As we’ve gone through the enumerator selection and training this week and a pre-test of the survey, a number of observations have come up – some related to firm surveys in particular, some more general. In no particular order:
Here is an idea – if you pilot a survey, then pilot the intervention (if you are going to) with the same folks that you did the survey with. And keep the contact information because then you can try and go back to these folks when you pilot the follow up survey. This ensures that you will have people to test the program related questions with (in my experience these get skipped in a lot of follow up survey pre-tests because they aren’t relevant to people who have never heard of the program).
One of the things that struck me as we did the training was the importance of enumerator body language. Watching the enumerators in practice interviews, the good ones were quite present with the respondent – things like cocking their head, making good eye contact. This is something that is important, but pretty hard to pick up in the job interview.
Is there a standard survey manual out there? Obviously a lot of the survey manual will be specific to the questions at hand, but there are some points that should always be there. For example, I may be showing my age, but I never thought about putting an admonishment not to use your cell phone during the interview in the manual. If you know of a resource like this that is on the web, send me an email – I will try to cover this in a future post.
Have enumerators practice actually filling out the questionnaire while still in training – and have someone walk around and watch them (the way we did this was to do a mock interview in front of them and have everyone record responses). This will make data entry easier.
When is a chair a chair? One of the things I love (although it simultaneously drives me mad) about debriefing after a survey pre-test is the existential nature of the discussion. In our recent discussions we spent much time (perhaps too much) discussing what is a chair. If the enumerator is sitting on a piece of wood that is destined as a productive input, is that a chair (remember, this is for a survey of small firms)? A toolbox? And then what is electricity – clearly a connection to the mains is. But what about solar or using car batteries? It turns out the main word in Chichewa for electricity is only connection to the mains – so alternate energy forms require more explanation. These are things that seem like the more straightforward part of the questionnaire, but hey that’s why we spend a lot of time debriefing.
In a related vein pretesting helps me understand a whole new set of potential mistakes. When I write questions for the questionnaire, I can usually guess which ones are going to be hard. And these clearly require a lot of training. But sitting through practice interviews – both in the training room and out in the field, I get a whole new sense of mistakes I didn’t think were possible (e.g. the transitive property isn’t obvious to everyone). I think some of this is country and even team specific – but it is humbling to be out there and see what can go wrong (and it also helps when it comes time for analysis).
What is privacy? We always stress that interviews should be done in private, with no one else around. For small firms, which operate long hours, it’s highly likely that you will have to do the interview at the place of business (this is particularly true for business women, who have a lot of other things to do when they get home). OK, but the successful business has customers (and this is something we really wouldn’t want the survey to drive away). If they are the drop in type, that’s fine (indeed, I subbed in for the shop owner in one interview – learning the different denominations of mobile phone air time). But what if the business is a hair salon? We had one interview where a woman sat quietly under the hair dryer for most of the interview. The enumerator was doing a good job of shooing other people away (friends, family members, hair salon staff who weren’t working) but clearly, waiting for this woman to finish what was a non-trivial hair treatment was going to be an issue – so we went ahead.
How do you know if someone has a document? This evaluation covers business registration, so we really care whether folks have a certificate. Alas, it seems like a lot of people in the pretest (some of whom we knew were registered) keep their certificate at home (they aren’t required to post them by any law). So one might suspect that the respondents are lying because they either want to please you or they think you work for the government (things you have to train the enumerators to avoid). But, one guy we were talking with gave a good explanation for this – during some fairly recent urban unrest, his shop had been damaged and looted. So home was safer. So how do we make sure this guy really has a certificate? Our approach is to allow for multiple codes in the response: yes has a certificate and shows it and yes has a certificate but doesn’t show. David and his coauthors Suresh de Mel and Chris Woodruff took a different approach in Sri Lanka. In that case the treatment was conditional on producing some documents, and for other documents they offered an incentive to respondents in terms of a chance to participate in a lottery when they were produced. An interesting idea, and one about which you would want to give advance notice of to respondents so you don’t have to come back.
Join the Conversation