Tolstoy's War and Peace was the big data of its time. A memorable moment from the epic novel occurs when Prince Andrei awakens following a severe injury on the battlefield. He fears the worst but, "above him there was nothing but the sky, the lofty heavens, not clear, yet immeasurably lofty, with gray clouds slowly drifting across them. 'How quiet, solemn, and serene, not at all as it was when I was running.'" Time appears to slow down and the Prince sees life more lucidly than ever before as he discovers the potential for happiness within him.
In many ways the scene captures what we demand of big data—not the bustle of zillions of data points as confusing as the fog of war, but sharp, clear insights that bring the right information into relief and help us connect strands previously unseen. The question of whether this idea is achievable is the starting point of a paper about big data on trade and competitiveness just published by the World Bank Group. In it, we asked—can big data help policy makers see the world in ways they haven't before? Are decisions that are informed by the vast amounts of data that envelop us better than decisions based on traditional tools? We didn't want a story trumpeting the miracles of big data; we wanted instead to see the reality of big data in action, in its messiness and its splendor.
The centerpiece of the report eventually became three pilots run last year by the Bank Group's Trade & Competitiveness Global Practice as a way of getting our hands muddy with big data. The first pilot used machine learning to make sense of government regulations (non-tariff measures in this case); the second pilot tried to assess the innovation capacity of cities through the data exhaust of its citizens, their networks, and the surrounding infrastructure; and the third tried to gather data about government response to specific types of economic behavior, in this case, cartel formation.
Through these projects we asked a variety of questions:
- What are the types of problems that big data can help solve better?
- What exactly is new about the data?
- How easy is it to access and use this data?
- What are the new tools and techniques to utilize this data?
- Are we and our clients/partners ready for big data?
- What works and what doesn't; where are the challenges?
The three pilot projects offered varying challenges, and we learned different lessons from each, as you will see in the report. The challenges centered on issues of data quality, data access, capacity to use the data, sustainability of big data initiatives, policy and regulatory uncertainty, security, privacy, data-philia and data-phobia. These correspond fairly well with those that we gleaned from the experience of other agencies that are described in the report in such areas as labor market monitoring using web data, trade analytics based on customs data, crowdsourcing of economic competitiveness measures, and better financial access facilitated by cell phones.
The study provides only a partial view of both the potential and pitfalls of big data. There are many emerging big data trends that are still not fully understood (we describe a few of them as well). The overriding message is that big data is already an ineluctable part of our lives and that both policy makers and the private sector must prepare for a world that is going digital faster than the next prediction about it. Trade, innovation, entrepreneurship, competitive policies, and our business environment are all areas that big data touches already and where there are major policy, capacity, and financing gaps even as entrepreneurs forge ahead with innovation products and services that clearly demonstrate the possibilities of big data.
It took a bludgeon on the head for Prince Andrei to see the world clearly. We should spare ourselves the pain!
The report was developed in a partnership between the World Bank Group's Trade & Competitiveness Global Practice (T&C) and the Innovations in Big Data program at the World Bank. Together, we worked on several activities, including workshops, events and pilots to put big data into action.
Access more than 2,000 trade and competitiveness indicators from more than 30 sources at TCdata360.
Join the Conversation