As social networks and search engines catalog our every click and keystroke, they generate billions or even trillions of data points about how and why we do the things we do. Companies trying to parse those disparate bits for clues about consumers have come up with a name for all that information – big data.
Broadly, big data is about using computers to search for patterns in huge amounts of information. And as columnists for The New York Times and NPR recently opined, 2012 was big data’s breakout year.
Big data played a huge role in the strategy and prediction of the election. It was lampooned in comics. It even became a meme among the kind of folks who get the wonky punch lines on “The Big Bang Theory.”
I started the year uneasy with big data. I was hearing the term everywhere, but had only the shallowest understanding of what it meant or why it was important. I had no idea if or how it might be relevant to covering medical studies.
Here’s the short answer: It is. And if you cover research, you need to understand it and be able to ask questions about it.
Geneticists already use big data in genome-wide association studies to find gene variations that may cause disease. As more doctors and hospitals convert to electronic health records (EHRs) the data those records generate will become huge pools of information that researchers can plumb to find risk factors for disease, effective treatments, etc. In January, MIT announced a new initiative to speed the use of big data in medical research.
One critical thing to understand about big data is that it’s not simply an additional tool that researchers can use to conduct studies. As Chris Anderson explained in a 2008 article for Wired, big data is a revolution in the way scientists think about information because it subverts the scientific method.
Instead using theory to drive the generation of information, researchers use raw information – and lots of it – to generate theories.
Depending on who you talk to, big data will either save medical research – freeing it from biases and other sources or error – or kill it, sending scientists down data-driven but ultimately meaningless blind alleys.
The latter is what recently happened to scientists who were using machine learning to understand big data in a Lancet study testing awareness in patients with brain injuries. More on that case study in my next blog post.
