Big Data - You Can Start Small

 So much data, so little evidence. So much information, yet unsatisfactory decisions...can big data change it, and how?

Big data is transformative and its potential limitless - we've all heard that and some of us even believe it. The big question though is 'how do you actually do big data' - how do you make use of its apparently limitless possibilities, especially if you are not an organization teeming with data scientists or if your organization was not born digital and your existing business model treats data as a traditional rather than disruptive asset?

This was the question that the World Bank Finances team posed to the two speakers at the 'Turning big Data into Big Impact' event at the World Bank on October 25, 2012. We wanted to know what we, as an organization committed to open data and open development, could do to better leverage the rapidly expanding data ecosystem around us. The number of open data sites grows daily; social media, sensors, mobile phones, satellites, et al provide continuously growing variety and volume of data; and big data related tools, applications, and technologies continue to become more accessible. Yet there lurks a sentiment that we need superior evidence that our work is having the requisite impact, and that we lack the tools to make course corrections and decisions when they still matter (rather than learn about them during audits and 'lessons learned' exercises after the time for meaningful action is past). How can we use big data to provide better evidence and make smarter and more timely decisions?

Jake Porway of DataKind, and Anthony Goldbloom of Kaggle spoke passionately, descriptively, and persuasively about, among other things, two specific tactical tools to test the waters - data dives and data competitions. Jake set the tone with a wide ranging presentation peppered with examples of data and data communities at work and spoke specifically about Data Dives which are weekend events that brings data scientists, topic experts, visualization experts, app developers, and others together to brainstorm data related issues around specific challenges. Think of them as hackathons for data scientists! They are, as Jake described them, an excellent vehicle for organizations trying to sort out what business challenges are riper than others for big data based approaches, what their data universe looks like, and what demand exists for potential solutions. Anthony followed with an insightful examination of Data Competitions in which' organizers present datasets and problems - data scientists from around the world then compete to produce the best solutions. At the end of a competition, the competition host pays prize money in exchange for the intellectual property behind the winning model'. Competitions can be a useful vehicle for organizations to not only tap into big data expertise from around the world but weigh different types of approaches and see measurable results.

A few themes emerged from the session --

1. Ask the right questions (data won't ask it on your behalf; people and their expertise matters a great deal)
2. Explore and experiment (but know what you are doing and why - and don't let the absence of standards deter you)
3. Look beyond your group/organization, and think multi-disciplinary teams
4. Big data skills cover a variety of areas - your organizations likely already has many of them
5. Think platforms, not just tools
6. Do different things, and do things differently

The biggest takeaway may have been this: big data isn't an esoteric, abstract concept that only applies to other organizations or requires skills that organizations such as the Bank don't have easy access to. There are simple techniques to introduce big data based approaches in organizations. The trick is to get started.

Event Storify - the event as it unfolded on social media

Thanks to Neil Fantom, Randeep Sudan, Aleem Walji and their teams for co-hosting the event. You made it all work!