When crises strike, policymakers need poverty data fast. But consumption surveys — the gold standard — are produced only once every seven years in low-income countries. Survey-to-survey (S2S) imputation offers a workaround: train on the most recent full survey, then score a short follow-up that collects 10–20 indicators. The result is a near-real-time poverty estimate at a fraction of the cost.
The approach works well — until a shock arrives. When COVID-19 lockdowns, conflict, or financial crises reshape household welfare, models trained in calmer times can go badly wrong. The intuitive response is to reach for more sophisticated machine learning. If linear regression can't handle shocks, surely random forests or neural networks can?
Our research shows this intuition is mistaken. The problem isn't the modeling algorithm — it's the information the model uses.
The core insight
S2S imputation breaks down during shocks for one primary reason: the variables used for prediction don't move when welfare does. Traditional predictors — housing quality, asset ownership, education, and household size — are slow-moving. When a shock hits, welfare can drop sharply while these indicators barely change. The model, seeing no movement in its inputs, predicts no change in poverty. The result is a large bias.
This failure affects all models in similar ways. No amount of algorithmic sophistication can recover a signal that the inputs never carried.
How proxy variables restore validity
Fast-changing proxy variables — whether a household consumed meat or sugar in the past week, or how they rate their economic situation — respond almost immediately to welfare changes. Including them lets the model "see" the shock. Our theory proves that when the proxy–welfare relationship remains stable, the augmented model produces reliable poverty estimates even under large shocks.
Evidence
Simulations
Monte Carlo results confirm the theory starkly. Omitting the proxy produces a 38-percentage-point bias — the model reports 46% poverty when the true rate is 84%. Adding a single fast-changing proxy eliminates the bias almost entirely.
Empirical evidence
We tested these predictions using paired household surveys from Uganda, Afghanistan, and Rwanda — three illustrative cases from a broader evidence base — and compared four models: Ordinary Least Squares (OLS), Random Forest (RF), Extreme Gradient Boosting (XGB), and Deep Learning (DL). All four are trained on the first round of data and scored on the second.
Uganda · COVID-19
- Poverty rose from 18.5% to 21.9% after the 2019/20 lockdowns. No-proxy models predicted a fall; with proxies, OLS recovered 21.2%.
Afghanistan · Severe crisis
- Poverty jumped from 38.3% to 54.5% (2011/12 → 2016/17). No-proxy models predicted 34–43%; with proxies, OLS recovered 53.3%.
Rwanda · Growth
- In Rwanda's 2005/6–2010/11 expansion, all four models performed well without proxies. When slow-moving indicators rise alongside welfare, the problem disappears.
The asymmetry
- Proxies matter most during negative shocks. In expansions, traditional indicators may suffice — and algorithmic sophistication adds little in either direction.
Practical implications
The message for practitioners is clear: invest in better data, not just better models. Linear models augmented with fast-changing proxies consistently performed as well as, and sometimes better than, complex machine-learning alternatives.
The priority should be collecting variables that respond to shocks: consumption indicators, food-security measures, subjective well-being, and short-term employment status. But proxy choice matters: inferior goods (whose consumption falls as income rises) can introduce bias. Screen proxies by checking that they move consistently with other consumption items and macro trends.
Validation is essential
No model is perfect. Model-based poverty predictions should be validated against other socio-economic indicators — GDP growth, non-monetary poverty measures, and food security conditions (see Lain et al. 2024). Evidence extends beyond this paper (e.g., Dang et al. 2026). But because dynamics vary across countries and episodes, we continue to expand the empirical base to 50+ countries and 100+ episodes.
Beyond poverty — jobs, refugees, social protection
The approach is already in use in and outside the World Bank. The Japan International Cooperation Agency (JICA) has applied it to measure the impacts of agricultural commercialization programs on income growth and poverty reduction, and is considering expanding its use. The same approach is being used to improve the shock-responsiveness of social protection programs and to monitor the living conditions of refugees.
Concluding remarks
When shocks disrupt household welfare, the binding constraint on poverty monitoring is informational, not algorithmic. No model can detect a change its inputs fail to register. The priority is building surveys that routinely collect shock-sensitive indicators alongside traditional predictors — modest in cost, transformative in use.
To obtain full information about the above discussion, see Yoshida, Kawashima, and Takamatsu (2026).
Join the Conversation