This post is coauthored with Ivette Contreras and Amparo Palacios-Lopez
The data points in a household survey tell the stories of how household members spend their time, whether they work or go to school, how previous experiences have affected their lives and how they cope with extreme events. However, these stories can be influenced by interview protocols, questionnaire design choices or by the person providing the information. It is why, our recent study[1] focuses on understanding how survey methods may affect the measurement of work, in particular for women and youth. Our work builds on existing evidence that has been summarized in previous blog posts (see here and here).
Measuring work accurately is crucial for policy making, especially in low- and middle-income countries, where the employment gap between women and men, as well as between youth and adults, is particularly large.[2] Even though gender- and age-based employment gaps can be explained by a variety of reasons, including gender norms and structural changes brought on by large-scale health and economic shocks, the estimated gaps are directly impacted by the quality and composition of the underlying data, specifically the potential undermeasurement of women and youth’s labor market outcomes.
Possible concerns with prevailing approaches to measuring labor outcomes in household surveys
There are at least two reasons that may explain why standard survey methods used to collect data on labor indicators potentially undermeasure women and youth’s labor market outcomes.
First, standard labor modules in household surveys are often better at collecting information on formal or regular activities but may fail to properly capture the informal and casual activities that are classified as work under current international guidelines.[3] For example, respondents may not consider their informal activities, such as preparing food to sell or helping in a family-owned business, as work.
Second, standard data collection protocols for household surveys allow for proxy respondents to provide responses on behalf of other household members. This practice may result in biased reports of labor indicators, particularly when the absent household member works in the informal sector and the proxy respondent is not aware of the household member’s labor activities (Bardasi et al., 2018; Kilic et al., 2023). This would be the case, for example, for a male proxy respondent who reports information on behalf of his spouse and/or young adult child living within the household.
A randomized experiment to reveal the effects of survey methods on measuring labor outcomes
To assess the causal impact of different survey methods in the measurement of work, we designed and implemented a randomized survey experiment in El Salvador. We were interested in understanding if changing survey methods may affect the results. First, we wanted to identify whether the provision of examples of common work activities may alter individuals’ responses about working, so we designed a List of Activities (LOA) module that included simple yes/no questions to assess if the respondent had worked over the past seven days. Some of the activities listed were helping a family member with agricultural work, selling clothes, preparing food items for sale, or providing transportation services.
Our hypothesis was that these questions may help improve the reporting of work by exposing the respondents to examples of income-generating activities that they could have undertaken in the last 7 days, with varying levels of labor/capital requirements, earnings potential and respondent involvement (in the context of work activities for household-owned enterprises), and in the process signaling that no activity should be deemed “too small” to count as work.
The design of this module was informed by the discussions held in focus groups organized as part of our research. For example, a woman stated she was not working. But when we asked for details about her use of time, she explained she usually spent six hours preparing tamales (a traditional dish in El Salvador) for sale, but because her husband was selling them in the market, she did not recognize this activity as work.
The second effect we explored was the proxy-respondent bias. Specifically, we investigated whether enforcing self-responses (instead of allowing for a proxy respondent) could change the measurement of individual’s work and employment status.
In sum, our experiment includes three groups:
· The first group completed a survey interview that included the LOA module before responding to the standard labor module. Proxy responses were allowed.
· In the second group, self-reporting was enforced for all eligible respondents in the standard labor module, but the LOA module was not included.
· The control group was made up of respondents who completed the standard labor module without the LOA. Proxy responses were also allowed.
Shedding light on the impact of survey methods
Our findings indicate that the List of Activities (LOA) module is more effective at addressing underreporting bias on labor market outcomes for women relative to men - the LOA module reduces the work gap between women and men by 8.2 percentage points (pp). We also observed that the LOA module increases the probability of women reporting a work activity by 8.1 percent when compared to women in the control group. In addition, we find that enforcing self-reporting reduces the employment and work gaps between young and older male respondents in the ESR group by 13.9 and 12.3 pp, respectively.
These effects are large and are highly relevant for survey practitioners, national statistical offices, researchers and policy makers that have a stake in accurate labor market statistics, especially on women and youth. The study provides evidence that data quality can be improved by including low-cost modules in household surveys as well by promoting the incidence of self-reporting when collecting individual-level data on topics that are crucial for the measurement of wellbeing.
----------------------------------------------------------
[1] This was a collaboration between the Living Standards Measurement Study, the World Bank household flagship program, and the World Bank’s Research Group
[2] For example, in 2022, only 43.8% of women compared to 67.9% of men were employed (ILO, 2022). Similarly, only 43.7% of youth between the ages of 15 to 24 years in low-income countries are recorded as employed compared to 71.3% of adults (ILO, 2022).
[3] The 19th International Conference of Labour Statisticians (ICLS) in 2013 introduced a new framework based on the first international statistical definition of work and a narrower definition of employment. More information about these definitions can be found here.
Join the Conversation