We are working on an evaluation of a large rural roads rehabilitation program in Rwanda that relies on high-frequency market information. We knew from the get-go that collecting this data would be a challenge: the markets are scattered across the country, and by design most are in remote rural areas with bad connectivity (hence the road rehab). The cost of sending enumerators to all markets in our study on a monthly basis seemed prohibitive.
Crowdsourcing seemed like an ideal solution. We met a technology firm at a conference in Berkeley, and we liked their pitch: use high-frequency, contributor-based, mobile data capture technology to flexibly measure changes in market access and structure. A simple app, a network of contributors spanning the country, and all the price data we would need on our sample of markets.
One year after contract signing and a lot of troubleshooting, less than half of the markets were visited at the specified intervals (fortnightly), and even in these markets, we had data on less than half of our list of products. (Note: we knew all along this wasn't going well, we just kept going at it.)
So what went wrong, and what did we learn?