An utterly fantastic long read by Christie Aschwanden at FiveThirtyEight.com, cuts to the chase very early: “Science is hard – really f***ing hard.”
Aschwanden’s piece – which ought to be required reading for every health journalist (and probably every news consumer, too) – aims to convey just how challenging it is to get reliable findings through scientific research and all the ways that science, despite our best efforts, is ultimately a human enterprise subject to human failings.
“If we’re going to rely on science as a means for reaching the truth – and it’s still the best tool we have – it’s important that we understand and respect just how difficult it is to get a rigorous result,” she writes. And of course, this reality is part of what makes responsible and thorough reporting on medical research such a challenge at times.
But reading her piece isn’t a challenge at all. In fact, it’s remarkable just how readable and accessible she makes truly abstract and dense concepts, leading the reader through an exploration of what the messy sausage factory of science looks like.
Aschwanden begins by acknowledging the various scandals that have plagued science lately, potentially undermining the public’s confidence in the whole truth-seeking enterprise. Then she very quickly illustrates how easy it is to manipulate data to get a statistically significant result with an interactive multimedia element on “p-hacking,” or changing the choices researchers would make in their data to achieve a p-value below 0.5, and therefore publishable. This mesmerizing, addictive p-hacker illustration reveals just how tenuous any “significant” results are. But she’s not making the point that scientists use statistics to lie or get what they want. Rather, “it’s much easier to get a result than it is to get an answer.” That’s true even for the most earnest of scientists.
It’s impossible to summarize everything Aschwanden covers in her look behind the curtain of scientific research. She discusses the shortcomings of p values and the biases that slip into researchers’ “degrees of freedom” as they try to analyze their data. She also explores the world of retractions with the help of Retraction Watch founders Ivan Oransky, M.D., and Adam Marcus. She explains the way biases taint every step of science and every way that we humans look at the world and interpret data. Even listing everything Aschwanden covers, however, would not convey how beautifully and effortlessly she captures the difficulty of doing good science and all the pitfalls awaiting even the most conscientious, ethical, truth-seeking scientists.
She gives us a tour of the forest but with multiple stops along the way to inspect individual trees. She helps the reader understand why those frustrating back-and-forth swings of scientific conclusions are precisely why science works as a self-correcting, if stumbling, truth-finding process.
Setting aside the time to curl up with this labor of love – Aschwanden spent months on her “obsession,” as she called it on Twitter – will undoubtedly influence the way you read studies, the way you regard scientists, the way you interview them and the way you report on their research. Even more importantly, by revealing all the ways in which science is hard and researchers can fail, Aschwanden somehow manages to restore confidence in just how amazing the process of science actually is.
