Tip Sheets

Are you covering a fakethrough?

Remember the burger grown from stem cells? It might be a great idea, except a single patty grown using today's technology, at least, cost a whopping $332,000.

Here, Jonathan Latham, Ph.D., the executive director of the Bioscience Resource Project, asks whether discoveries like that are breakthroughs or "fakethroughs" – scientific advances that will never progress to new treatments or beneficial products. He also talks about his brand of investigative science journalism and why reporting on new discoveries should probably be more muted.

By Jonathan Latham, Ph.D.

In a recent article, we used the word "fakethrough" to describe genetically engineered crops that never progressed to humanitarian products. The trouble is that many of these "firsts" were robustly covered in the media, where they often became the basis of major coverage and big-budget promotions. 

In our style of investigative science journalism, we focus on ideas. With ideas you cannot just follow the money, so we imagine what the world would look like if that idea didn't exist.

If the world would look better for ordinary people, (e.g. without genetic determinism in Are Genes for Disease a Mirage?) and certain major interests would suffer, then we examine that idea very closely for hype and supporting evidence. 

In the course of doing this work, we have come to appreciate  strongly the idea that the conclusions of a scientific experiment only apply to the conditions under which it was performed. Thus experiments on male rats don't necessarily apply to female rats, or to rats fed a different diet, or rats of a different strain, or mice - or humans, for that matter.

The usefulness and necessity of this rule of thumb is implicit in how scientists keep their methods tightly controlled and consistent and describe them precisely, and it's also why other labs don't always get the same results – apparently tiny variations in protocols can make a big difference to the results (and so we would suggest their generalizability must be doubted).

We take this rule of thumb equally seriously when we interpret results. It means we don't assign any (there are some exceptions) significance to animal experiments (including "humanized" animals), except to that species itself. Nor do we assign any significance to experiments that rely on tissue culture, whether it is a layer or 3-D cultures, because there are too many artifacts, as well as contamination problems. In our experience, if experiments in another species or in tissue culture, mirror what you expect, there is a high probability it's just by chance.

You will note that this approach excludes a large proportion of the research carried out by the National Institutes of Health and the National Science Foundation. The modern convergence of researchers on a few reductionist methodologies of questionable utility (but high convenience) indicates not progress but rather how little thought most scientists put into the limitations of their experiments, especially when everyone else is using them. But for us the value is that it explains the more important problem of why the "war on cancer" (and every other disease) has hardly/not at all progressed. In our view, the sooner overly reductionistic research is ignored by the media (because it is preliminary), the sooner it will go away. Research, after all, is largely driven by public demand, as NIH officials occasionally admit.

So, what should a journalist report?  Journalists should focus on researchers who look at real-world conditions and long term impacts; ignore proxy measurements; are careful enough to define their problem sets accurately; and don't say things like, "I can't define consciousness, but we all know what it means." It also means covering stories where human/mice/tissue culture results disagree, and not treating them as aberrations (as The New York Times did recently).  Instead, these results should be treated as scoops that question – and in fact disprove – parts of the whole dysfunctional edifice. 

So here are two tips I would suggest to any reporter covering an advance in medicine or science:

• Focus on a discovery's limitations, rather than simply its potential.

• Think about what the experts or researchers don't want to talk about, not what they do.

Further reading:

David Horrobin (2003) Modern biomedical research: an internally self-consistent universe with little contact with medical reality? Nat Rev Drug Discov. 2: 151-4.

Jonathan R. Latham, Ph.D.., is co-founder and executive director of the Bioscience Resource Project and editor of the Independent Science News website.