This blog is part of “AI for Data, Data for AI”, a series aiming to unwrap, explain and foster the intersection of artificial intelligence and data. This post is the second installment of the series—for further reading, here are the first and third installments.
Two independently evolving data innovations for high frequency monitoring of commodity prices in fragile contexts
The first two Sustainable Development Goals (SDG1 and SDG2) focus on eradicating poverty and achieving zero hunger. Monitoring and tracking progress towards these goals require innovative analytics and input of comprehensive data, including commodity prices (Andree et al., 2023). Across many low- and middle-income countries (LMICs), commodity price data is often rife with gaps or sparsely updated due to limited infrastructure and budgetary resources to monitor market functioning and assess price changes on time.
Recent advancements in data innovation offer promising solutions to address the paucity of data and guide food security intelligence for diverse development stakeholders. Two major independent approaches are gradually evolving to generate high frequency price data and fill data gap across countries. On one hand, the imputation of prices with artificial intelligence (AI), using machine learning algorithms, is helping to address data gaps by nowcasting and forecasting price trends. The World Bank Development Data Group commenced an initiative to generate commodity prices in real-time with AI-model, across 36 countries (See Andree B.P.J. 2021; Andree B.P.J and Pape U.J., 2023). These data are available for public access and use through the World Bank data repository (RTP).
The AI-based price estimates provide valuable insights into relative food price dynamics and valuable input into food security outlook assessment. However, it is also crucial to validate the accuracy of these estimates by comparing with independently submitted ground truth prices, to gain more credibility and encourage further use by stakeholders.
An opportunity to meet this need is emerging through the collection of high frequency price data submitted by wide network of volunteer citizens (“crowd”). This approach has been applied (in various ways) to collect data on food prices across different national contexts, providing valuable information on market conditions and trends. Between 2019 -2021, a cohort of three (3) research partners piloted an initiative to crowdsource daily prices of major food commodities in the core northern region of Nigeria achieving daily dataflow across several key commodities and market segments (See Adewopo et al., 2021; Solano-Hermosilla et al., 2022; Arbia et al., 2023; and FPCA).
Validating AI-estimated price data with high-frequency ground truth prices
Beyond inference on indicative price signals, two (2) major questions are pertinent for further application of these innovative data methods. First, are ground-level crowdsourced price data comparable to enumerator-collected ground truth prices? Second, are AI-estimated real time price (RTP) data correlated to ground-level crowdsourced prices over time? To answer these questions, we assessed the correlative relationship between AI-generated and crowdsourced food prices in core north-central region of Nigeria, with focus on three (3) states where data was available: Kano, Kaduna, and Katsina.
This region accounts for extensive grain production in the country, but historically marked by religious crisis, terrorism, banditry, and relatively low literacy. We analyzed price data for two major staple grain commodities, maize and imported rice, using monthly AI-estimated prices and the daily ground truth price data.
The AI-estimated prices accurately tracked the submitted prices, within statistically acceptable margins (Figure 1a & b). Notwithstanding nuances of commodity sub-type, packaging units, and market segments, the near-perfect agreement between price data trend from both sources provides a first real ground-truth assessment of WorldBank's AI-generated RTP inflation estimates.
Figure 1. Comparison of submitted ground truth maize grain prices† from crowd volunteers (“crowdsourced”) and trained enumerators over an eight-month validation period, juxtaposed with monthly AI-estimated price spreads at four markets in the northern region of Nigeria.
(a) The chart illustrates difference in data volume and inherent variability, providing insights into the reliability and patterns of the combined dataset.
(b) Comparison of daily-averaged maize grain prices† from volunteer citizens (“crowd”) and trained enumerators over an eight-month validation period, harmonized with monthly AI-estimated OHLC prices* in the northern region of Nigeria. The chart demonstrates the synchronized co-variation among the three independent sources, highlighting the cohesion of price signals at daily timesteps. † Based on prices of 100Kg packaging Units. *OHLC Prices – Open, High, Low, and Close Prices.
Demonstrating application in fragile contexts
The value of both innovative data methods transcends mere understanding of current or past trend of commodity prices. Since consumers are constantly making decisions on the quantity and quality of commodities to purchase based on market prices, real time monitoring of food prices is valuable for assessment of food affordability and outlook for household-level food security.
For instance , the AI-approach has been used to generate high frequency accurate prices of fresh produce in Papua New Guinea (Andree and Johann, 2023), while the crowdsourced price data provided one of the first near-real-time evidence of inflationary effect of COVID-19 pandemic on prices in low- and middle-income countries (Adewopo et al., 2021).
Our recent Food Security Report shows that inflation is a lingering threat to livelihoods globally, and this is projected to worsen in many countries. Timely access to market price data is important to assess shocks and deploy right-fit interventions, especially in fragile, conflict, and violence (FCV) contexts where majority of the population are vulnerable to food insecurity and have limited or no access to social safety nets. It is also a significant milestone towards forecasting emergent food crises based on the richness of dataflow to test various assumptions about price movement, investment outlook, and overall resilience or vulnerability of households to shocks and stressors (as proposed by Andree et al. 2020).
Data collection with a mobile phone by an assigned enumerator, following conventional survey procedure in a market within the Northeastern region of Nigeria. Photo: IITA
As both innovative approaches co-evolve, the findings from this study establishes instructive evidence to support further development of these methods towards bridging extant data gaps in food price monitoring systems within low- and middle-income countries, and contribute to advancement of data-driven decision-making at regional, national, or sub-national levels.
Read more about the data validation assessment insights here, or visit the open access data repository to access, query, and use the AI-generated monthly prices for Nigeria (here), or the high-frequency ground truth data (here).
Join the Conversation