Syndicate content

Survey Data

Data quality in research: what if we’re watering the garden while the house is on fire?

Michael M. Lokshin's picture

A colleague stopped me by the elevators while I was leaving the office.

“Do you know of any paper on (some complicated adjustment) of standard errors?”

I tried to remember, but nothing came to mind – “No, why do you need it?”

“A reviewer is asking for a correction.”

I mechanically took off my glasses and started to rub my eyes – “But it will make no difference. And even if it does, wouldn’t it be trivial compared to the other errors in your data?”

“Yes, I know. But I can’t control those other errors, so I’m doing my best I can, where I can.”

This happens again and again — how many times have I been in his shoes? In my previous life as an applied micro-economist, I was happily delegating control of data quality to “survey professionals” — national statistical offices or international organizations involved in data collection, without much interest in looking at the nitty-gritty details of how those data were collected. It was only after I got directly involved in survey work that I realized the extent to which data quality is affected by myriad extrinsic factors, from the technical (survey standards, protocols, methodology) to the practical (a surprise rainstorm, buggy software, broken equipment) to the contextual (the credentials and incentives of the interviewers, proper training and piloting), and a universe of other factors which are obvious to data producers but usually obscure and typically hidden from data users.

If development data is so important, why is it chronically underfinanced?

Michael M. Lokshin's picture

Few will argue against the idea that data is essential for the design of effective policies. Every international development organization emphasizes the importance of data for development. Nevertheless, raising funds for data-related activities remains a major challenge for development practitioners, particularly for research on techniques for data collection and the development of methodologies to produce quality data.

If we focus on the many challenges of raising funds for microdata collected through surveys, three reasons stand out in particular: the spectrum of difficulties associated with data quality; the problem of quantifying the value of data; and the (un-fun) reality that data is an intermediate input.

Data quality

First things first – survey data quality is hard to define and even harder to measure. Every survey collects new information; it’s often prohibitively expensive to validate this information and so it’s rarely done. The quality of survey data is most often evaluated based on how closely the survey protocol was followed.

The concept of Total Survey Error sets out a universe of factors which condition the likelihood of survey errors (Weisbeg 2005). These conditioning factors include, among many other things: how well the interviewers are trained; whether the questionnaire was tested and piloted and to what degree; whether the interviewers’ individual profiles could affect the respondent answers, etc. Measuring some of these indicators precisely is effectively impossible—most of the indicators are subjective by nature. It may be even harder to separate the individual effects of these components in the total survey error.

Imagine you are approached with a proposal to conduct a cognitive analysis of your questionnaire. - How often were you bothered by the pain in the stomach over the last year? A cognitive psychologist will tell you that this is a badly formulated question: the definition of stomach varies drastically among the respondents; last year could be interpreted as last calendar year, 12 months back from now, or from January 1st until now; one respondent said: it hurt like hell, but it did not bother me, I am a Marine... (from a seminar by Gordon Willis)

Predicting perceptions of WBG Sectoral Support: Can Country Opinion Survey data help?

Svetlana Markova's picture

The World Bank Group collects and analyzes feedback of its stakeholders systematically—on a three-year cycle—in all client countries. The Country Opinion Survey data helps the institution better understand local development context, enhance stakeholder engagement and partnerships, and improve its results on the ground. Tracking stakeholder opinions about effectiveness of the Bank Group’s sectoral support over time is crucial for assessing the progress on top development challenges in countries.

We continue looking at the education sector, as an example, to see how the Country Survey data can help country teams predict stakeholder perceptions and improve effectiveness of Bank’s sectoral work. Let’s look at the trending data, explore what drives the change in numbers, what can be projected for future and why, and how to work with stakeholder perceptions, using a targeted approach.

The first chart shows how the World Bank Group’s stakeholders—partners from the Government, ministries, civil society, private sector, and donor community—have changed their views on the Bank’s work in education in the Central Asian countries—Kazakhstan, the Kyrgyz Republic, Tajikistan, and Uzbekistan—during the past years.[1]