Getting big data in shape for the development marathon (part 1)

If you are confused by all the buzz about “big data” and how it’s going to rock your business or industry, rest assured you are part of a teeming crowd. 

Few businesses – running the whole gamut from one-man start-up to global behemoth – have actually successfully integrated any “big data” opportunity into their core offerings or operations.

On one hand, the concept is presented as a technical challenge: because we are swimming in such a sea of digital sensors – from the internet to CCTVs – huge amounts of data are being collected, way beyond the capacity of conventional computer methods to process and present properly in a useful way.  

On the other hand, “big data” is sold as a big break for virtually all businesses; with the right APIs and interfaces, everyone can tap into this massive “global datastream” – an “uber-consciousness” in every respect – and extract valuable patterns and insights with massive commercial implications.

The trouble is that while processing capacity has improved tremendously, the methods are still pretty crude for handling unstructured data (which is the form in which most real-world information is actually available) seamlessly and intelligently, notwithstanding what Autonomy’s bosses might tell you. 

Nor are the aforementioned APIs and interfaces up to scratch.

“Big data”, it would appear, is, sadly and conventionally speaking at least, in poor shape at the moment to significantly impact your business, much less transform it.

But what if we were to be more modest about the concept and practical implications of “big data”? 

Rather than talk about petabytes and zetabytes and such giant magnitudes, what if we stay with gigabytes but apply the kernel of insight at the core of big data, which is that “data torrents, well-configured, can bring wisdom”?

Some observers believe that the big data concept, if trimmed down into “big data lite” so to speak, so that humans and machines can work jointly at and with it, can yield a concept of analysis that can improve perception and intuition, and even upturn entrenched “stock wisdom”, replacing such with true enlightenment. 

There is a considerable range of applications for big data lite and a whole array of hallowed domains waiting to be disrupted if only more attention could be paid to the gigabyte scale in the big data spectrum.

Think of the interstices of census and inter-censal data; the small print annexed to consecutive national financial budgets; and assorted qualitative results from major longitudinal studies of sociological or psychological contexts. Think of better handling, manipulation, visualization and representation to enable a more intense “crowd-review” of such information output.  

This intermediate domain, considerably below the threshold of “big data”, may yet unveil its most precious gems. For example, large-scale review of court records (including hard-to-access flavours like juvenile criminal records) is revealing amazing sociological patterns about the deeper structure of society with implications way beyond law and order. More fascinating, such data is being interposed with genealogical records to construct what can best be called “moving maps” of social evolution, unbelievably striking in depth and colour (of interest: http://www.weforum.org/reports/big-data-big-impact-new-possibilities-international-development).

Now think of the mass application of “big data lite” to hitherto badly stereotyped emerging market phenomenon. Think of all the prospects of disruption in the dominant narratives of global development. In my next piece, I will illustrate my meaning.

Bright B. Simons is the President of MPedigree Network, Ghana and a member of the Global Agenda Council on Information & Communication Technologies and a 2012 Young Global Leader of the World Economic Forum.