Beyond Big Data

I wish policy makers and funders of research would visit this site which rightly, in my view, attempts to move us collectively beyond the current preoccupation with 'big data'.

Here is the associated press release:

CAMBRIDGE (Dec 17) — One question Big Data can’t answer: how do we know what’s important in a complex world?

Big Data keeps getting bigger, whether in biology, medicine, social sciences or social media. But how do we know which pieces matter in continually larger piles of data? The trick is to recognize patterns in the largest scale of behavior. These patterns, determined by relative handful of information, are not only the key to understanding a system, but they also tell us how we can influence its behavior going forward.

In an article released today, complex systems scientist Yaneer Bar-Yam, president of the New England Complex Systems Institute, explains how his team has been successful using this approach to predict and explain market crashes, food prices, the Arab Spring, ethnic violence, and other complex biological and social systems.

Rather than amass larger and larger data sets, determining which information is pivotal (and ignoring the rest) is the key to solving the world’s increasingly complex challenges.
The article, titled “Beyond big data: Identifying important information for real world challenges” describes a general way to understand complex systems.

“Big data analytics is a buzzword today,” Bar-Yam explains, “but the key to using any data is understanding its significance, and this requires knowing the patterns in the data and how we can affect or change the system’s behavior. This requires key ideas about the way the system functions.”
Complexity first made its mark in physics by demonstrating how traditional approaches of statistics and calculus fail to describe such radical transitions as the moment water boils into vapor. These transitions required a new kind of mathematics. Bar-Yam and his group have been applying generalized versions of the new mathematics to answering real world problems.

“Understanding the problems we face in the world today requires more than just data, it requires insight into radical changes that will take place in the future,” Bar-Yam stated. “We have to understand transitions that might happen in the behavior of systems, not just how they were behaving yesterday.”

I would argue that investing in Systems Thinking in Practice is needed to focus on the end of the graph where what is important needs to be understood and acted upon.