Ghost protocols: Scientists propose a way to plug major holes in the medical literature

Reporters who cover medical studies often take great care not to be fooled by the spin put on research by drug companies, universities and even government agencies.

But sometimes the spin is the study itself, and that’s a serious problem. It’s a big story that’s hiding in plain sight, to borrow a phrase Steven Brill likes to use.

By some estimates, half of clinical trials are unpublished. Half. And because positive studies are far more likely to be published than negative studies – a phenomenon called publication bias – the studies that don’t get published often throw some seriously cold water on how good a treatment looks.

If this research hasn’t been published, how do we know it exists? Some of these trials have been released because of lawsuits; others can be found in standardized documents called clinical study reports that drug companies file with the FDA and its counterpart, the European Medicines Agency. Regulatory agencies use them for their reviews, but because they’re never published in medical journals, they remain hidden to the medical community and general public.

For a case in point, consider the antidepressant reboxetine. Unpublished studies that were brought to light in a stunning 2010 meta-analysis in the British Medical Journal showed that Pfizer had failed to publish data – all of it negative – on 74 percent of patients who had participated in the clinical trials of the medication.

“Not only does the drug not work, it really doesn’t work,” wrote the blogger SciCurious in a guest post for Scientific American on the reboxetine revelations.

Hidden or unpublished clinical trial data also is at the heart of the recent questions about the effectiveness of the drug Tamiflu.

“This is a systematic flaw in the core of medicine,” said Dr. Ben Goldacre, a British physician and a leading scientific watchdog, in a 2012 TedMed talk. “This is a disaster. We cannot know the true effects of the medicines we prescribe if we do not have access to all the information.” Goldacre also is a driving force behind the AllTrials campaign, an Internet petition that aims to raise awareness about the issue.

Various remedies for the problem have been tried with limited success. Starting in 2008, the FDA Amendments Act required reporting of all clinical trials of drugs, devices, and biologics with at least one study site in the United States within a year of the study’s completion.  But a 2012 study in the BMJ found that only 22 percent of the trials subject to the new regulation have complied with the law.  A separate analysis, published in Pediatrics, found some companies were shutting down trials with negative results early in an apparent attempt to skirt reporting requirements. Proposed legislation clarifies that the law applies to trials regardless of their outcome. Well, duh. But thanks anyway, D.C.

Pharmaceutical companies have also repeatedly pledged to police themselves when it comes to reporting all their data.

Now some of the same researchers who have been working to release unpublished Tamiflu data think they may have found a fix. If studies are missing or misrepresented in the medical literature, they say outside authors should be able to publish or correct that research. In a paper and editorial published last week in the BMJ, the researchers dub this concept RIAT, for Restoring Invisible and Abandoned Trials.

In a sense, they’re calling for medical ghost writers, except these authors would get credit for their efforts and wouldn’t be paid by the trial sponsors. They’ve laid out a detailed plan for how outside authors can responsibly publish another researcher’s work.

RIAT could solve problems on several fronts. Young investigators, who are often under intense pressure to publish to further their careers, could find plenty of readymade data to dive into in these orphaned or ghost studies. That might help stem the rising tide of scientific fraud and could perhaps also put a dent in the raft of weak and unreliable data that’s picked up by predatory or junk journals.

“RIAT is about cleaning up the scientific evidence,” said Peter Doshi, Ph.D., a postdoctoral fellow in comparative effectiveness research at Johns Hopkins University School of Medicine in Baltimore, Md.  “Really, I hope this can start to set a new standard,” he said.

Doshi and his co-authors say they have already identified more than 178,000 pages of unpublished data on common drugs such as Relenza, Seroquel and Plavix. In some cases, these are not small, inconsequential studies. Studies that remain unpublished include the largest trial ever conducted on Tamiflu and the largest study of the drug gabapentin, sold under the brand name Neurontin, for diabetic neuropathy. In other cases, studies that have been published appear to be incomplete because they don’t include missing data.

So far, three medical journals, the BMJ, PLoS One and PLoS Medicine, have signed on to the idea and agreed to consider RIAT manuscripts, provided that authors first follow certain steps designed to give the original researchers a chance to set the record straight themselves.

The initiative is important, the journal editors write, because “public confidence in the credibility of medical research is at a low ebb.”

So keep an eye out for RIAT studies. Let’s hope enterprising authors embrace the concept and take advantage of the offers made by these journals to publish the work.

Because when we’re only getting seeing half of the studies, we’re only getting half the story.

5 thoughts on “Ghost protocols: Scientists propose a way to plug major holes in the medical literature

  1. Adam Jacobs

    “They’ve laid out a detailed plan for how outside authors can responsibly publish another researcher’s work.”

    I’m not sure I agree with that statement. I would rather say they have laid out some vague aspirational dreams about how outside authors can responsibly publish another researcher’s work.

    What is crucially missing from their proposal is any hint of who’s going to pay for it. Papers don’t write themselves. Until someone figures that out, I suspect all these unpublished papers will remain unpublished.

    It’s certainly true that there are a huge number of clinical trials done in the past and never published, and the world of medicine would be a far better place if they could see the light of day. However, I think we need more than fine words if that’s ever going to happen.

  2. Pingback: AHCJ members contribute tips on best uses of | Association of Health Care Journalists

  3. Brenda Goodman

    Thanks for your comment. You raise an important point. I, too, wondered about the issue of funding while I was reading over the RIAT proposal.

    Certainly, it would require plenty of time and effort to do the hard work of preparing the data for publication. And who is going to pay for that?

    Perhaps universities who pay professors and assistant professors to teach and research would accept work on a RIAT paper as an acceptable part of those investigators professional duties, and the work could be funded that way. Perhaps in the future government grants could help fund some of this work as well.

    I think the idea is worthy, though, and could work given a bit of support.

  4. Adam Jacobs

    Well, it could be done that way, but that would leave all those professors with less time to do other (presumably useful?) things, so even if it could be done that way, there would be enormous opportunity costs.

    For me, it’s hard to see where the money comes from. I guess the choices are that it comes from taxpayer money or it comes from the pharmaceutical companies. I guess the ideal would be if the pharma companies paid for it, but whether they’d be willing to do so without being made to do so by new regulation is another question. But I guess it wouldn’t be too hard to make it a part of new regulation: perhaps as a condition for keeping already licensed drugs on the market.

    As for taxpayer money, well, that’s not exactly in over-supply at the moment, so it’s hard to see that happening.

    There’s also the question of whether you pay (expensive) professors to do the work, or whether you hire professional medical writers. I suspect the latter could be considerably more cost-effective (declaration of interest: I run a company providing medical writing services).

  5. Pingback: Drug company takes a stand in fight for clinical trial transparency | Association of Health Care Journalists

Leave a Reply