Published on Development Impact

Improving satellite data for economists: a new approach to regression estimation

This page in:
Improving satellite data for economists: a new approach to regression estimation

Measurement errors in Earth Observation (EO) data and their implications for its use in economics are becoming a theme of my writing here. But it bears repeating that as EO data becomes easier to use it is also becoming easier to use without fully understanding the potential flaws within the data. These often arise because many of the products that are actually used, which is not to say all of the available EO data, are not direct measurements. They are predictions and it is relatively easy to embed systematic measurement error from the prediction process directly into our outcome or treatment variables while still having low standard errors.

Many people have tried to develop solutions to the challenge of how to parsimoniously account for the measurement error inherent in these products when using them in regressions. A new paper proposes an interesting new approach that does not require making assumptions about the structure of the error in the data generation process. 

The Problem: Prediction is Not Measurement

Earth observation (EO) map products, including both classification (e.g., forest/non-forest) and continuous maps (e.g., percent tree cover), are among the most widely used inputs in applied work across development, environmental, and urban economics. These products are typically not direct measurements. They are model-based predictions trained on limited, and sometimes biased, ground truth, with accuracy that varies across space, time, and land cover.

A common practice is to extract these map values and use them in regressions. That can induce bias. If a map overestimates cropland in one area and underestimates it in another, and those errors are correlated with your outcome or covariates (say, rainfall or elevation), coefficient estimates can be systematically off. And because map errors are often non-classical—directional, spatially structured, and heterogeneous—treating them as simple i.i.d. noise is usually wrong. This applies to both binary and continuous EO products.

There are methods in the EO and statistics literatures to “correct” maps, but most impose strong assumptions about the error process (e.g., specific forms of misclassification or parametric measurement-error models). Those assumptions are rarely justified in the low-data settings where EO products are most useful.

A New Solution: Prediction-powered inference (PPI)

The method described in the new paper is called Prediction-powered inference, or PPI. At its core, it’s a statistical correction that debiases map-only regression coefficients using a probability-sampled ground-truth holdout. Usefully, PPI doesn’t require assumptions about the underlying data generation process or its error structure. Instead, it uses a holdout sample of ground truth data, drawn from the same population as the map itself, to calibrate the map-only estimator without assuming a specific error structure.

The idea is simple. Imagine you have a cropland map of a region (“satellite data”), and a random sample of ground points where you know whether cropland was present or not (“ground-truth”). You can estimate the relationship you’re interested in using the ground-truth data. This will be unbiased but generate large confidence intervals. You could also estimate the relationship using the satellite data. This will generate a precise estimate but might be biased. PPI corrects the map-only coefficient with a calibration term from the ground-truth subsample and constructs confidence intervals via a percentile bootstrap.

Unlike approaches that adjust for uncertainty only in standard errors or confidence intervals, PPI adjusts the point estimates themselves. That means it can correct for bias, not just account for variance. And it does so in a way that requires no knowledge of the classifier or training process used to generate the original map. That makes it broadly applicable. One especially useful feature is that it can be applied equally to error-in-Y (your remotely sensed data is the outcome), error-in-X (your remotely sensed data is one of your regressors of interest), or error-in-both regimes. The key requirement is that there is at least some ground-truth data available.

 

Image

 

The paper demonstrates the usefulness of this method by applying it to several real-world examples that demonstrate different combinations of error-in-Y, error-in-X, or error-in-both relationships. 

In all four cases the authors examine the satellite-only estimates are significantly more precise than either the ground-truth or PPI adjusted estimates. In some cases, they are also substantially biased, but in inconsistent directions (i.e. sometimes larger, sometimes smaller than the true coefficient). In all cases the PPI adjusted estimates yield confidence intervals that are either closer to the true coefficient or whose confidence intervals overlap the true coefficient, or both. 

The Value of Ground Truth

One of the key contributions of this paper is its recommendation for how EO products should be released going forward. The authors argue that map producers - those generating these classification products - should release a probability-sampled holdout ground-truth set (unused in training/tuning) with sampling details alongside the map. This isn’t the same as publishing training accuracy metrics or confusion matrices. It means providing a sample of the raw, independently collected ground truth labels, so that downstream users can calibrate the predictions themselves.

If that’s not possible the authors suggest that users collect their own. Even a small number of visually inspected points from high-resolution imagery can be enough to improve inference, especially if drawn using a transparent sampling process.

Most economists don’t want to (or can’t) create their own EO data from the raw imagery. But we can and should think carefully about how to validate and adjust the EO data we use. And producers of these datasets should recognize that their maps are being used in regression applications, not just for visualization or summary statistics.

Having a standardized holdout set would enable better corrections and help avoid downstream misuse. It would also push the EO and economics communities a bit closer together, encouraging more collaborative work and better communication of uncertainty.

What Economists Should Take Away

If you work with satellite data this paper should make you rethink how you use them in regressions. They should not be treated as observations. They’re modeled estimates, and these processes have their own errors and mistakes. Using them as if they were true can bias your results, especially in policy-relevant settings.

What PPI offers is a way forward: a statistically principled, empirically grounded method for adjusting inference when using predicted data. It won’t solve all the problems with EO data, but it gives us a much better baseline than assuming away error.

We’ve come a long way in using satellite data to answer questions that would be difficult to answer in its absence. The next step is making sure we’re using it correctly.


Patrick Behrer

Economist, Development Research Group

Join the Conversation

The content of this field is kept private and will not be shown publicly
Remaining characters: 1000