How well can you plan your survey: the analysis of 2,000 surveys in 143 countries
This page in:
Our interviewers are still in the field, we need more time to complete the survey, could you extend our server for two more months? We receive such requests every day. Why do so many of our users fail to estimate the timing of their fieldwork?
Survey Solutions is a free platform for data collection developed by the World Bank and used by hundreds of agencies and firms in 143 countries. Many users of the Survey Solutions host data on free cloud servers provided by the World Bank. A user requests a server by filling in a form where he indicates the duration of the planned survey, the number of cases to be collected, and provides other relevant information. We impose no restrictions on how long a user can use the servers. Any survey end date is accepted. Over the last six years we have accumulated data on more than 2,000 surveys. We use information about surveys that collected 50 or more cases for this analysis.
How well can people conducting surveys follow the survey schedule?
Survey duration
Figure 1 shows differences between the survey end dates anticipated by our users and the dates registered in server activity logs. 15% of users finished their surveys on time (within a week of the date they planned the survey would end); about 47% of users underestimated the amount of time they needed for the field work (right tale, positive difference in days), and the rest finished surveys faster than they initially planned (left tale, negative).
The losses for agencies conducting surveys are higher if the survey duration is underestimated or if a survey started later than expected. The contracts with interviewers and other actors involved in the fieldwork need to be extended, the interviewers have to be paid for the extra days; the activities that depend on completion of the survey work are shifted, etc. Agencies that finish surveys earlier could also incur some losses, but these are relatively small.
Are people getting better at estimating field work duration the more surveys they conduct? Figure 2 shows that compared to novice users, a higher proportion of users who already conducted at least one survey finished their fieldwork earlier than they reported on the survey request form. But the proportion of users who underestimated the duration of their fieldwork declined only marginally between these two groups.
A user can extend her server if the fieldwork spans past the original survey end date. Figure 3 presents the distribution of the number of requests for server extension for novice and experienced users. About 43% of our users extended their servers more than once, with about 3% requesting 4 and more server extensions. That means some of these users, after underestimating the timing of the survey, repeat the same mistake again and again.
Similar to Figure 2, the assessment of fieldwork duration improves with experience: the share of users who requested only one extension increased from 54% for the new users to 63% for the “experienced” users. However, even among the latter group, about 12% of users revised the end date of their survey three or more times over the course of their fieldwork.
Number of questionnaire revisions during the field work
Another dimension of a proper survey planning is the timely preparation and testing of the survey instruments. CAPI or CAWI surveys simplify the process of questionnaire updates and allow addressing issues undetected during the piloting when the survey is already in the field. For example, the analysis of incoming data might show that some questions take respondents too long to answer, indicating that such questions could be badly formulated. If so, an updated version of the questionnaire with better wording for the problematic questions could be deployed. This, however, comes with a cost of potential loss of data comparability. The best practices of survey management recommend a finalization of the questionnaire before the fieldwork and minimizing the number of questionnaire revisions once a survey is in the field. This is especially important for surveys conducted digitally because, compared to paper questionnaires, digital survey instruments usually contain validation conditions and algorithmic skip patterns that should be carefully tested before the field work.
Figure 4 depicts the frequencies of questionnaire revisions during the fieldwork for all surveys in our sample and for users who conducted two or more surveys. About 20% of our users have used a single version of the questionnaire during their fieldwork. Most of the users revise their survey instruments more often. A surprisingly high proportion of novice users made 10 or more revisions of their questionnaires during the fieldwork. The number of questionnaire revisions declines for more experienced users, but the proportion of users revising their instrument 6 or more times while in the field still seems to be high.
Conclusion
We present here a unique (to our knowledge) analysis of metainformation about a large number of surveys conducted in virtually all developing countries. It is difficult to qualify our results because we are missing the baseline to which to compare our statistics. The survey fieldwork depends on many factors that are beyond the control of people conducting surveys. Political unrest, changes in donor funding policies, delays caused by bureaucratic procedures, etc., all could derail the survey schedule. More formally, we have no information about the actual variances of statistics we are presenting. Theoretically, we could calibrate, for example, the ability of our users to predict duration of their fieldwork or the number of times they modify questionnaires, by similar statistics from the best surveys firms or most advanced national statistical offices. Unfortunately, we have no such information. Nevertheless, we do believe our findings highlight important issues that data collection agencies need to think about during the survey planning processes.
Good quality surveys are expensive. An average nationally representative survey in developing countries costs about US$2M (Kilic et al. 2017). A large part of these costs could be incurred as payments to interviewers, transportation costs, and other time-sensitive expenses. Closely adhering to a survey schedule seems to be critical for managing costs of the surveys and ensuring high data quality. Our findings demonstrate that many agencies involved in data collection are not able to plan their fieldwork properly. This could result in overblown budgets, negatively impact of the quality of collected data, and delay the release of data to the public. The survey planning agenda is often neglected in the capacity building efforts. The international development community might need to rethink the survey training curriculum as the returns on improvements in the ability of data collection agencies to plan and follow survey schedules appear to be high.
Very interesting data and analysis Misha and Zurab.
On the survey duration:
The variance of days under/overestimated strikes me as quite large. Under/overestimating survey duration by 100 days would imply quite poor planning indeed. A few more factors may be driving the observed difference:
The specified end date may differ from the end date users expect in their own plans/logistics: When filling the form to request a server some users may already add a buffer. Possible reasons include a) just in case, b) there are no direct costs to the users, c) the message in the date question that the server will be deleted 30days after the specified date, or d) users might want time for cleaning post field work. One would expect more experienced users to do this more as they might have learnt from past experiences to factor in a contingency. This may explain some of the surveys finishing earlier than expected.
Data cleaning on server: After the main field work has finished, often there is a small tail of activities on the server, such as supervisors/HQ still having some requests or questions, adding comments, rejecting, a few interviewers trying to chase the last non-responses. These activities may cause the survey to appear delayed in the meta data, while the activities have been factored into the budget/timeline of a survey, or affect them only marginally.
Field work delay vs extended duration: As you rightly mentioned it in the conclusion there are factors external to survey management that cause delays of the start date and hence the end date of a survey, such as changed election dates, insecurity, weather, funding not coming through, delays from counterparts, etc. Such factors would affect the timing, but not necessarily the (field work) duration of a survey, and might require different conclusions. Of course, delays are also often due to insufficient planning, e.g. the national holiday that was found out about last minute and delays training and field work, insufficient time allocated to preparation activities, lack of contingency during rainy season, etc.
On the questionnaire versions:
While I have seen quite a few cases where updates to the questionnaire during field work were due to insufficient testing or ill-conceived questionnaires and indicative of bad planning or preparation, there are also other reasons that point in the opposite direction:
Logistics: In some field work set-ups it helps using multiple versions of an unchanged questionnaire to organize field work. For example, in a recent survey that is fielded over a one-year period we made new versions of the same questionnaire in regular intervals simply to ease data management.
Improving vs fixing: If done well, updates during field work can further improve already well tested and functioning instruments while preserving comparability. Examples of such updates are codifying frequent other specify values or to adding checks or instructions to parts of the questionnaire where interview errors have been identified and addressed. This requires a certain level of data management capacity. Seeing only one version of a questionnaire for a survey could mean a really well tested and functioning instrument, but likewise a lack of capacity to interpret and react in a timely manner to incoming data and to fix or improve the instrument.
While the above points may influence some of the results, your analysis still clearly shows that planning or implementation according to schedules is generally poor. I agree with the conclusion that adding survey planning (and implementation!) to the survey training curriculum would be very useful to improve on the cost effectiveness, timeliness and quality of data collections in the development sector.
Andreas,
Thanks much for your insightful comments. The points you are making are valid. We also realize that in some cases there are legitimate reasons why a survey fieldwork runs longer than expected. However, we still see a large number of cases, and we have more anecdotal evidence, where surveys are just badly planned and implemented.
Thanks
In my experience as a field manager in data collection, the main factors which may impact the survey turn around time may be the selection of capable interviewers with interest and involvement in the subject, close monitoring and mid-core correction plans will be the main triggers.