When Is A Baseline More Than A Baseline? 5 Uses for Your Baseline Survey

|

This page in:

So you’ve designed an awesome impact evaluation, you’ve carried out a rich baseline survey, you’ve presented the baseline results to the government of Brigadoon, and now you….wait two years until the follow-up survey? What else can you do with this baseline data? You can do a lot! You can write a report, you can write a brief, you can publish papers, you can test targeting strategies, and you can even [drumroll] affect policy.

Use 1: Write a report. The baseline report is a useful way to bring all the results into one place. A recent baseline report for an impact evaluation of community-based education in Afghanistan has interesting descriptive data on school attendance, demand for education, and learning levels. A baseline report for an impact evaluation of agribusiness leaders in Ethiopia has data on business practices, why women start businesses, and financial habits. One for a cash transfer program in Malawi has useful data on household livelihoods and adolescent attitudes toward sex.

Of course, many of these come in at 100+ pages (or 350 pages across two volumes, in this extremely data rich example on education in Tanzania). Almost no one – including your government counterparts – is going to read your full report. Another downside of the baseline report (by itself) is that because a wide array of information is in one report (usually creatively entitled “baseline report”), there is no way for anyone to know that your report contains insightful data from the inside of Afghanistan’s classrooms or on the financial habits of Ethiopian entrepreneurs.

Use 2: Write a brief (or two). In the hopes that these data will be consumed, you can write one or more briefs, summarizing the most interesting descriptive data in 2-4 pages. For example, this three-page brief from the baseline of an IE of a nutrition and maternal health project in the Gambia summarizes data on the attitudes toward family planning and stockouts of family planning supplies. Another brief from the same baseline survey summarizes data on antenatal care and health worker knowledge.


The nice thing about the brief is that because it’s just a few pages, you can share it much more easily. The downside is that the distribution mechanism, as for baseline reports, isn’t well established. You can hand the brief to people, you can e-mail it to people, and you can tweet about it, but it’s still unlikely to come up if someone is searching for data on family planning in the Gambia.
 

Use 3: Write a paper. An informal survey of papers (many thanks to colleagues and to Twitter) reveals that it is indeed possible to publish papers based on baseline surveys. These have the advantage that (1) they tend to be more focused than a general baseline report, and (2) they are in journals, both of which make them more findable to people who are working on the topic.

What do papers based on baseline surveys look like?
  1. Use novel descriptive data to illuminate an interesting phenomenon. For example, Banerjee et al. used a weekly absenteeism survey in health centers, along with other rich data, to characterize the relationship between health, wealth, and health services in one part of India. Gibson et al. used a survey of remittances to calculate the first estimates of the cost-elasticity of remittances. Lee et al. demonstrate a surprising fact, that electrification among households in rural Kenya are extremely low, even when they are in the vicinity of an existing electrical grid.  
  2. Combine data from multiple baseline surveys. Naudeau et al. combined data on children’s cognitive development (as associated with socio-economic status and other variables) from a baseline in Cambodia and a baseline in Mozambique. McKenzie and Siegel drew from list experiments across 4 baseline surveys in 4 countries to elicit illegal migration rates. McKenzie and Woodruff asked similar questions on business practices across 7 baseline surveys and examine the relationship between business practices and business outcomes.
  3. Baseline Plus. Some studies use the baseline plus some additional information. For example, Akresh et al. used two rounds of a survey in Burkina Faso – but before any IE results – to examine child ability and human capital investment decisions. Diwan et al. and de Mel et al. used baseline data plus program uptake data to examine how the invitation structure for business training in Kenya or how the offer of wage subsidies affected the take-up of these programs.
  4. Add an experiment (or a thought experiment) to your baseline survey. De Martino et al. carried out a lab-in-the-field experiment in the course of a baseline in Brazil. Jayachandran included a series of hypothetical questions about preferred sex composition of children at different fertility levels. Increasingly common electronic survey administration makes tests of questionnaire design easier.
Does anyone publish or cite these papers?
Across 11 published baseline-based papers that I identified that have been accepted for publication, I found examples in the American Economic Review Papers and Proceedings (1 and 2), PLOS One, Economic Development and Cultural Change, Public Finance Review, Management Science (forthcoming), Development Engineering, Migration Studies, Pacific Economic Bulletin, Global Health: Science and Practice, and as a book chapter. So the median outlet is a field journal, with a few exceptions.

Across the 7 papers that had come out by the end of 2014, the Google Scholar median citation count is 22, which is lower but not radically lower than recent citation counts in top economics journals, per Card and DellaVigna. The average is higher (50), which compares very favorably to an economics-wide average published recently.

Use 4: Inform program design. One team I spoke with is using the baseline survey to figure out the best way to target future iterations of the program: Baseline data can be used to figure out which subsets of variables are most predictive of a larger, richer set of household characteristics.

Use 5: Affect policy. Even at the baseline stage, data from the impact evaluation can affect policy! David McKenzie has written about how, in response to his paper on the cost-elasticity of remittances for Tongan migrants to New Zealand, “This spurred an effort by the New Zealand Government and World Bank operational team to work to reduce the cost of sending remittances, leading to a change in a law in New Zealand that had limited one option for sending money cheaply and the development of a website to share information on the costs of sending money.” Likewise, the baseline for an irrigation impact evaluation in Ethiopia showed that “female headed-households lag male headedhouseholds with regard to technology adoption, agricultural yields, assets, and income,” and as a result, “the government agreed to pilot strategies for improving women’s access to and take-up of new agricultural technologies.”

Conclusion. Once you’ve finished the PowerPoint presentation of the baseline data, don’t stop. Ask yourself, can these data and results be useful to a larger audience than just the government counterparts I’ve presented them to? If so, write that paper.

I’d love to see other examples of papers that came out of baseline surveys, or of other uses for baseline surveys, in the comments!

Bonus Reading

Authors

David Evans

Senior Fellow, Center for Global Development

Climent Quintana-Domeque
March 03, 2016

More examples
Dear David,
Many thanks for your very interesting post.
We would like to draw your attention to our 2 papers from a project on the evaluation of the effects of (randomly) paving (residential) streets in Mexico. Our first paper was based on the Baseline Survey (2006), while the impact evaluation (our second paper) was based on both the Baseline (2006) and Follow-up (2009) surveys.
In the first paper, we provide an analysis of the reliability of homeowners' estimates of the value of their houses, as obtained through a household survey: The Reliability of Self-reported Home Values in a Developing Country Context (Marco Gonzalez-Navarro and Climent Quintana-Domeque) Journal of Housing Economics, 2009, Vol. 18, No. 4: 311-324. http://www.sciencedirect.com/science/article/pii/S1051137709000412
In the second paper, we provide the first experimental estimation of the effects of the supply of publicly financed urban infrastructure on property values: Paving Streets for the Poor: Experimental Analysis of Infrastructure Effects (Marco Gonzalez-Navarro and Climent Quintana-Domeque), Review of Economics and Statistics, forthcoming. http://www.mitpressjournals.org/doi/abs/10.1162/REST_a_00553
Best wishes,
Climent and Marco

Geetha Nagarajan
March 09, 2016

Nice blog to underscore the many uses of baseline findings, Thanks, David.
We recently did a couple of policy briefs using our baseline findings. They got great traction with stakeholders in Malawi (donors, implementers and the Government) in terms of intense discussions and immediate changes to make the programs work better. You can find the briefs here:
a. Beyond Access: Improving Service Utilization in Malawi
http://pdf.usaid.gov/pdf_docs/pa00kksp.pdf
b. Implementing Integrated Development in Malawi
http://pdf.usaid.gov/pdf_docs/pa00kksq.pdf
- Geetha Nagarajan
Social Impact, Inc.