How should we measure food security during crises? The case of Nigeria
This page in:
High-frequency monitoring of access to food has become especially important during the recent COVID-19 pandemic. Poor access to food can have both short- and long-term impacts on health and wellbeing and is thus an important targeting criteria. Furthermore, many simple food security indicators have been shown to be extremely sensitive to income and price shocks, so changes in such indicators can be used as rough proxies for changes in monetary poverty and other multidimensional metrics of wellbeing.
Measurement of food access during crises increasingly relies on a consensus of different indicators. However, during the recent pandemic, there is a growing list of surveys that rely on a single experiential indicator of food access—the Food Insecurity Experience Scale (FIES). Part of the reason for the extensive reliance on this indicator is its inclusion as an official measure for tracking progress towards Sustainable Development Goal 2 (zero hunger), and the fact that nearly 100 countries have either incorporated the FIES into their national statistics or are in the process of doing so as of 2019.
Despite the growing popularity of the FIES, the evidence validating experiential indicators has been mixed[1]. Data from the Nigerian Living Standards Survey (NLSS), collected immediately prior to the COVID-19 pandemic[2], afford us the opportunity to compare the FIES to other common food access metrics and monetary poverty in the specific but important case of Nigeria. We demonstrate that the FIES does not appear to correctly identify the food-insecure population in the country.
First, a large share of the population identified by the FIES as being food insecure is not monetarily poor. Figure 1 reports the share of the population that is food insecure by consumption decile. There are roughly equal shares of the population in each consumption decile that are identified by the FIES as having poor food access, which includes the richest income deciles that likely suffer from very little food insecurity.[3] Importantly, the same figure demonstrates that the Food Consumption Score (FCS), which is one of the primary indicators used by WFP to target food assistance, primarily identifies the poorest households as having poor or borderline food access.[4]
Figure 1. Poor or borderline FCS values are much more prevalent in the lower deciles of the monetary consumption distribution, but this is less clear for severe food insecurity as per the FIES
Additionally, spatial variation in the FIES was also not aligned well with poverty and other indicators of food access. Figure 2 demonstrates that food insecurity using the FIES was more prevalent in the south than in the north.[5] However, the same figure demonstrates that monetary poverty and poor food access using the FCS was significantly more prevalent in the north of the country than elsewhere. Importantly, a wide range of other sources corroborate the pattern demonstrated by both monetary poverty and the FCS, where poor food security is far more prevalent in the in the north of Nigeria than elsewhere.[6]
Figure 2. Food insecurity is more prevalent in the south according to the FIES, while the opposite is true for the FCS
Unfortunately, it is unclear exactly why the FIES is not aligned with other measures of wellbeing and food security. One possibility is that many of the questions on which the FIES is based are subjective, such as whether households worried about food consumption, and might have a tenuous link to actual food consumption. However, Figure 3 demonstrates that the striking north-south differences appeared in all of the FIES’ constituent questions, and not just the more subjective questions that prior research has demonstrated might be difficult to interpret. More investigation is going to be needed to determine precisely why the FIES does not match other indicators.
Figure 3. Food insecurity appears to be more prevalent for all 8 FIES questions in southern Nigeria, when presenting results at the zone level
Despite the inability of the FIES to identify the vulnerable populations in levels, it is possible that the measure can do better at detecting changes in food access. and recent projections suggest that this dual crisis could leave an additional 10 million Nigerians in poverty by 2022. Consistent with this projection, Figure 5 demonstrates that the FIES is able to detect a worsening of food access in Nigeria in high-frequency data collected throughout the COVID-19 crisis. However, given the discrepancies noted above, it is unclear whether the increase in poor food access is being driven by households that actually have poor food access.
Figure 4. The prevalence of moderate and severe food insecurity as per the FIES has risen in 2020
Given significant discrepancies between the FIES and other indicators of wellbeing in Nigeria, particularly in the regional concentration of poor food access, high-frequency monitoring of food access should consider other metrics as well. In particular, the FCS, and other metrics of dietary diversity, tend to be particularly sensitive to income and price changes and are potentially a better option to measure food access in a crisis. Additionally, , and the potential causes of the decline. In particular, employment changes in the same high-frequency surveys can further identify regions and households with sudden changes in income, and food price databases can further help to identify regions where food supply chains have been significantly impacted by the pandemic.
This blog was updated on December 1 to further clarify that these findings are based on Nigerian data.
[1] Here are a number of articles collected by FAO that investigate the relationship between the FIES and household characteristics.
[2] The comparison uses data from the Nigerian Living Standards Survey (NLSS), collected between September 2018 and October 2019. The survey includes modules necessary to estimate the FIES, other common food access metrics (e.g., the Food Consumption Score), and monetary poverty.
[3] Households are classed as severely food insecure if their ‘raw score’ – the sum of all 8 questions in the FIES module – is 7 or 8. This is a tenable approach if the data satisfy the basic requirements of the Rasch model, which is the case for the 2018/19 NLSS, where the infit statistics for each of the 8 FIES elements range from 0.85 to 1.15 and the Rasch reliability statistic is 0.77.
[4] Poor or borderline food access corresponds to those with FCS values less than or equal to 42.
[5] These results do not appear to stem solely from using the raw score instead of the Rasch model to calculate the prevalence of food insecurity. When applying the Rasch model separately at the zone level with the RM.weights package in R, severe food insecurity still appears to be more widespread in the South South and South West zones than in the North East and North West zones.
[6] Additional sources include Famine Early Warning System Network’s (FEWS NET) Integrated Phase Classification (IPC) Food Security Projections and the WFP situation reports.
Brilliantly apt, insightful and well projected.. Great thanks and God bless your endeavours endlessly
Interesting work Jonathon, Sharad and Tara - thanks for undertaking this. In case you hadn't seen it already, you might be interested in some related comparative of methods work (albeit more on poverty targeting than food security detection) by Pascale Schnitzer (also at the WB):
https://openknowledge.worldbank.org/handle/10986/13084
Dear Jonathan, Sharad and Tara,
Thanks for the analysis you present.
I definitely agree with you that more investigation is going to be needed to determine precisely why the FIES does not match other indicators, as this is a quite peculiar result with this specific dataset in Nigeria, that we have seen only once in the hundreds of other datasets we have processed since the FIES was introduced in 2014.
I wonder: before comparing them, have you equated the FIES-based measures across the Northern and the Southern regions? These are so different, both culturally and linguistically, that they be better treated as two separate countries for FIES purposes.
Also, have you explored the association between FIES score and decile of real consumption separately by zone? This may help detecting whether the lack of association in the expected direction is specific to some of them.
I would definitely like to see more of these analyses, conducted on the many datasets the Bank holds.
It would contribute to the important on going debate on how to feasibly and effectively measure food insecurity, something I have been very involved with over the last decade.
(see for example https://nyaspubs.onlinelibrary.wiley.com/doi/10.1111/nyas.12594, https://www.sciencedirect.com/science/article/abs/pii/S0263224117307005, and chapter 17 in https://www.routledge.com/Food-Security-Policy-Evaluation-and-Impact-As… )
Yours,
Carlo
Thanks so much Carlo for taking the time to read the blog and for these very useful comments. We totally agree that this is going to need more investigation for the case of Nigeria.
On the north-south issue, we tried a few different methodologies when preparing the analysis for the blog. As well as calculating the FIES raw score and looking at each individual FIES question, we also implemented the Rasch model for each zone separately (using the RM.weights command in R), to try and treat the inter-zone comparisons like inter-country comparisons, having received some very useful guidance from colleagues both inside and outside the World Bank. However, using the Rasch model did not substantially change the results, with severe food insecurity appearing to be more widespread in the South South and South West zones than in the North East and North West zones, so the puzzle we are finding remains.
In a sense, this issue is similar to the one we face when trying to calculate monetary poverty for Nigeria – we impose a national poverty line then deflate consumption spatially to compare with that national standard, rather than setting zone-specific poverty lines. For this piece, we thought it was important to try and do something similar for food security, so we could make statements about the entire country rather than just within specific zones (the generalizability versus specificity trade-off!). Nevertheless, your suggestion to check what might be happening across deciles within zones is certainly something we will look at as we try and disentangle why these patterns arise for Nigeria.
We also are keen to expand these types of analyses to additional countries and to further our understanding of how best to effectively measure food insecurity. Looking forward to continuing the discussion!
Many thanks
Dear Tara/David,
One analysis I would encourage you to conduct is to look at the share of raw score zeros over the total, by State. You will discover some very suspicious results, such as Adamawa revealing the highest share of cases where all responses have been coded as "no". As - as far as I can tell - this is one of the poorest states in the country, these many cases where all answers are "no" may point to various potential issues. First, there may have been language/cultural barriers, with poor, less educated people having more trouble understanding the questions and therefore answering "no" by default. Second, there may have been simply problems with some enumerators who may have filled the questionnaire with all "no" without truly asking these questions (unfortunately, this is not an unheard problem, and doing some quality check on the distribution of raw scores received by enumerator may reveal interesting facts).
The main technical issue here is that you won't be able to detect any of these problems by simply looking at the results of the Rasch model, as the extreme cases with raw score zero or raw score 8 do not contribute information on the relative ranking of the items, and therefore do not enter in the model's estimation.
Third and last, there may be the issue of the need to properly equate the scale obtained in different regions, as I mentioned in my comment, especially when, as it is the case in Nigeria, linguistic and cultural differences are marked. Have you done that? I'd be curious to learn which conclusions did you draw.
Feel free to reach out directly at [email protected] or at [email protected] if you need any kind of assistance in conducting these analyses.
Thank you very much for reaching out again. These are all important points, so just responding to each of them in turn:
1. Yes, you’re right to say that there were a number of households in Adamawa where all questions were recorded as a “no.” However, several southern states also had a relatively high proportion of households responding “no” to all of the FIES questions. Because of this, there aren’t dramatic differences between the north and south in the prevalence of all zeros at the zone level. Additionally, these same households that responded “no” to all questions answered a number of more complex questions in ways that one would expect in a large multi-purpose survey, suggesting that the complexity of the questions may not be a problem per se. Lastly, we thought it might be difficult to uniformly throw out responses that are potentially informative.
2. Data quality may always be an issue and it is something that we thoroughly check before we use household surveys to construct welfare measures. In many careful checks, we found the quality of this particular survey to be high. Our colleagues in the Development Data unit were thoroughly involved in all aspects of data collection and can more thoroughly speak to the training of enumerators and the exact data quality checks that were used in the data collection.
3. When we applied the zone-level Rasch model, we used the R code posted by FAO online (RM.w and equating.fun). Our understanding is that the second of these steps is what you’ve mentioned above. Even after doing this, both severe and moderate food insecurity are more prevalent in the South South and South West zones than in the North East and North West zones.