Tolstoy's War and Peace was the big data of its time. A memorable moment from the epic novel occurs when Prince Andrei awakens following a severe injury on the battlefield. He fears the worst but, "above him there was nothing but the sky, the lofty heavens, not clear, yet immeasurably lofty, with gray clouds slowly drifting across them. 'How quiet, solemn, and serene, not at all as it was when I was running.'" Time appears to slow down and the Prince sees life more lucidly than ever before as he discovers the potential for happiness within him.
In many ways the scene captures what we demand of big data—not the bustle of zillions of data points as confusing as the fog of war, but sharp, clear insights that bring the right information into relief and help us connect strands previously unseen. The question of whether this idea is achievable is the starting point of a paper about big data on trade and competitiveness just published by the World Bank Group. In it, we asked—can big data help policy makers see the world in ways they haven't before? Are decisions that are informed by the vast amounts of data that envelop us better than decisions based on traditional tools? We didn't want a story trumpeting the miracles of big data; we wanted instead to see the reality of big data in action, in its messiness and its splendor.