Data to decisions

We are at the beginning of the Big Data era, and there is widespread anticipation that this will be a huge benefit to companies. A recent MIT report shows that companies that practice “data-driven decision-making” have on average a 5% better profit margin than similar companies with more traditional decision-making processes.

In our “Data to Decisions” session at the World Economic Forum Annual Meeting in Davos, we heard chief executives tell us how Big Data can reinvent everything from customer relationship management (CRM) and internal processes to product design.  We also heard that there are significant challenges in data sourcing, permission agreements, data quality and, of course, privacy concerns; most Big Data is personal data about customers. Fortunately, these challenges can be addressed by conventional business practices.

There is, however, a more subtle but even bigger problem: how Big Data turns the process of decision-making inside out. Normally, you approach a problem with some understanding and a hypothesis about what to do. Then you construct a test of your hypothesis to validate it; statistics tell you true or false. The power and danger of Big Data is that you don’t need to start with a good guess; you can “data mine” patterns from the data. But these patterns are often beyond human understanding and can depend on context in unexpected ways, as we saw during the 2010 “flash crash” and during the 2008 derivatives meltdown.

Because normal statistical methods of decision-making and validation fail us when we turn to Big Data, we have to seek validation through Living Labs. These can be the sort of rapid, real-world consumer testing which companies such as Amazon or Nike use, or the special trade zones where governments test new rules of commerce.

The lesson is that Big Data can dramatically improve your company, but you have to match Big Data with real-world testing.

Alex Pentland, Toshiba Professor of Media, Arts and Sciences, Massachusetts Institute of Technology (MIT), USA; Global Agenda Council on Information & Communication Technologies

Picture: Binary code background, Shutterstock