A growing trend towards digitization, digital storing, processing, and managing of information, as well as the spread of electronic devices and electronic communication lead to generation of high volumes of data. “Big data” collected from sensors, mobile devices, and computers create new opportunities, as well as new challenges for the way organizations operatee.
To illustrate the new opportunities opened up by big data, IBM reports that 2.5 quintillion bytes of data is created on a daily basis, and almost 90% of all data was created within the past 2 years. McKinsey predicts that the volume of global data will grow at the rate of 40% annually. A company like Walmart, processes over 1 million transactions per hour and owns over 2.5 petabytes of data. According to Gartner, developing big data capability and technology will account for $28 billion of IT spending in 2012 and will increase to $34 billion on 2013. Now, most of the big data spending goes towards social network and content analysis. Many organizations have already started developing core competencies related to big data and are customizing their existing solutions to the new environment influenced by big data. Governments are also facing new challenges of creating an enabling environment for businesses to be able to operate with large amounts of data, especially containing sensitive information.
1. Big data is increasing security and privacy awareness. Governments, citizens, as well as companies and clients show greater sensitivity towards data and its privacy. As noted by Cloudera CTO Amr Awadallah speaking at Wharton BizTech conference, people and organizations are more willing to give up control even over their financial resources by, for instance, putting them into bank accounts, than to give up control over their data. In December 2012, California Department of Insurance announced that it was launching investigation of Nationwide Mutual Insurance Co which reportedly had its computer system penetrated. The potential number of affected individuals whose personal and financial data, including tax returns, social security numbers was accessed, could reportedly reach over a million. Security of computer systems and data is becoming key to preserving assets and reputation of organizations.
2. Big data is improving transparency and efficiency of operations. Data-driven decisions lead to better efficiency and opening up non-sensitive data helps make organizations more transparent. Open data movement has shown how data stimulates innovation and development of analytical capabilities. New applications and data visualization have made services and information more accessible and help better engage citizens and clients. According to David Lee, Director of Product Development at Box, speaking at Wharton, most of the data used by citizens is created trough desktop applications and specialized content apps.
3. Big data is driving infrastructure development. Growth of data will stimulate organizations to rely on cloud providers to store the data. Gartner predicts that by 2016, nearly 1/3 of the data will be stored in the cloud and the cloud industry will double. The US was among the first governments to switch to cloud computing. The government launched FedRamp program which aims to accelerate secure cloud adoption and authorize the IT services which are used by government agencies. For scientific and research purposes, big data is creating the need for powerful machines to process it. The US government owns 251 of 500 top supercomputers. Oak Ridge National Laboratory in Tennessee managed by the US Department of Energy has the world’s most powerful computer Titan with 17.59 Petaflop/s (quadrillions of calculations per second) and 560.640 processors. Europe has 105 out of 500 top super computers, China – 72, No 2 country after the US.
4. Big data is creating jobs. According to indeed.com, there are over 17.000 jobs in big data immediately available in the US. Gartner predicts that by 2015 this number will increase to 1.9 million jobs. Globally, 4.4 million jobs will be created to support big data. McKinsey estimates that by 2018, US will experience a shortage of 140.000 to 190.000 of analytics professionals and almost 1.5 million of big data professionals, from analysts to managers. To address the shortage of talent, companies, for example EMC, started to offer training and certification programs in big data. Universities are launching Master’s program in analytics, very large information systems and big data to prepare next-generation specialists.
Join the Conversation