A cautionary tale: Have you checked that citation?

Tara Haelle

About Tara Haelle

Tara Haelle (@TaraHaelle) is AHCJ's medical studies core topic leader, guiding journalists through the jargon-filled shorthand of science and research and enabling them to translate the evidence into accurate information.

Photo: Tom Walker via Flickr

No fewer than four of the 2016 winners of the Awards for Excellence in Health Care Journalism covered the opioid epidemic from different angles. No surprise — former Surgeon General Vivek Murthy noted last year that more Americans suffer from addiction than from cancer. During his tenure, Murthy made the opioid crisis a top priority.

Murthy’s 2016 report on addiction has been compared to an influential report on smoking decades earlier, and he created a website devoted specifically to addressing the opioid crisis. But how much of it could have been prevented with a bit less complacency on the part of researchers? It’s an obvious question to ask after reading a thought-provoking (and somewhat maddening) research letter in the June 1 issue of the New England Journal of Medicine.

The research that the four Toronto researchers submitted was simple enough for an undergraduate to undertake in a few hours, but no one did until now. Their conclusion is startling:

“We found that a five-sentence letter published in the Journal in 1980 was heavily and uncritically cited as evidence that addiction was rare with long-term opioid therapy. We believe that this citation pattern contributed to the North American opioid crisis by helping to shape a narrative that allayed prescribers’ concerns about the risk of addiction associated with long-term opioid therapy.”

The 1980 letter read as follows:

“To the Editor: Recently, we examined our current files to determine the incidence of narcotic addiction in 39,946 hospitalized medical patients who were monitored consecutively. Although there were 11,882 patients who received at least one narcotic preparation, there were only four cases of reasonably well-documented addiction in patients who had no history of addiction. The addiction was considered major in only one instance. The drugs implicated were meperidine in two patients, Percodan in one, and hydromorphone in one. We conclude that despite widespread use of narcotic drugs in hospitals, the development of addiction is rare in medical patients with no history of addiction.”

The letter was signed by Jane Porter and Hershel Jick, M.D., of the Boston University Medical Center Surveillance Program.

From 1980 through the end of March 2017, that letter was cited 608 times, with a sudden jump — citations more than doubled — in 1996. That was the year Oxycontin entered the market. The vast majority of these citations, 72 percent of them, cited the letter as evidence that opioid pain therapy rarely led to addiction – which we know today couldn’t be further from the tragic truth. Further, 81 percent of the citations did not mention that the letter references hospitalized patients, and some of the statements “grossly misrepresented the conclusions of the letter,” the NEJM authors wrote. Consider some of the examples they provide in their appendix:

  • “This pain population with no abuse history is literally at no risk for addiction.” (1998)
  • “Fear of addiction may lead to reluctance by the physician to prescribe. […] However, there is no evidence that this occurs when prescribing opioids for pain.” (2002)
  • A 1994 citation presumes, with no evidence, that the letter was referring to pain in cancer patients.

My personal favorite was published in 2001: “Although medicine generally regards anecdotal information with disdain (rigorously controlled double-blind clinical trials are the “gold standard”), solid data on the low risk of addiction to opioid analgesics and the manageability of adverse side effects have been ignored or discounted in favor of the anecdotal, the scientifically unsupported, and the clearly fallacious.” The letter did not report results from a “gold-standard” randomized controlled trial and could hardly be called “solid data.”

As recently as 2014, authors in a study about morphine use for cancer patients wrote, “In reality, medical opioid addiction is very rare. In Porter and Jick’s study on patients treated with narcotics, only four of the 11,882 cases showed psychological dependency.” Even the word “study” here is a stretch. The letter offers no information about methods, no details about the population or the dosage, no limitations, and no references to other studies into opioid risks. It is as though no one else in medical research decided to see if opioid painkillers might cause addiction.

Certainly, studies did eventually accrue looking at addiction risk, but it seems confirmation bias, laziness or complacency – or all three – conspired to point researchers toward this letter as a toss-away citation to dismiss the risk so they could move along to whatever point they were making. Thus a brief letter published nearly four decades ago may have helped justify opioid prescribing patterns contributing to the current crisis.

To be fair, much of the blame about this letter is supposition as well. A problem as complex as opioid addiction does not spring from a single cause. OxyContin’s manufacturer and executives did not plead guilty to federal charges in misrepresenting their drug’s risk for no reason, and the authors do not discuss other possible contributing factors.

However, the salient point for journalists is to not trust that a list of citations at the end of a study actually supports what the authors claim. (I discovered this myself recently in a story I wrote about American Academy of Pediatrics recommendations.)

It is not possible or reasonable for a journalist — especially on deadline — to check every reference in a study. But do keep an eye out for statements that warrant some skepticism and offer no elaboration other than the superscript number of the citation, especially in the introduction or discussion. Too often, the cited study does not say what the citing authors claim it says and can be outright irrelevant. I have had researchers point out such inappropriate citations when I have interviewed them about potential red flags in a study I am covering.

In a publish-or-perish academic world, it unfortunately is not surprising that researchers may cut corners like this in their papers. One thing journalists can do is not let them get away with it.

Update: Doctor Who Wrote 1980 Letter On Painkillers Regrets That It Fed The Opioid Crisis

4 thoughts on “A cautionary tale: Have you checked that citation?

  1. Paul Burke

    Great point. When a citation “seems too good to be true, it probably is.” I check it and its methods before quoting either study, and usually find big flaws. Most recently I saw Yale’s estimates of opinion in each Congressional district about global warming, based on modeling from mostly national surveys. They haven’t published any validation for areas this small (700,000 people), so we don’t know how big their errors are.

  2. Pat Anson

    The 1980 NEJM letter wasn’t that far off. Rates of addiction among people who are medically prescribed opioids are fairly low. The NIH estimates only about 5% of patients taking opioids as directed for a year end up with an addiction problem.

    It all depends on how you define addiction, misuse, abuse, etc. In a review of 38 studies, one researcher found a wide spectrum of addiction rates among pain patients, ranging from 0.7% to 34.1%. Of course, its the upper range of that estimate that is repeatedly cited by the CDC and others who want to blame patients and doctors for the overdose epidemic.

    https://www.painnewsnetwork.org/stories/2015/3/31/study-claims-10-of-pain-patients-addicted-to-opioids

  3. Jennifer Palter

    Tara, did you quote that five-sentence 1980 letter exactly? The final sentence has two tenses of the verb “to be”–“is” and “are,” with no connecting slash (/), and then is missing the most important word, which I suppose from the context should be “rare”:

    We conclude that despite widespread use of narcotic drugs in hospitals, the development of addiction is are in medical patients with no history of addiction.

  4. Pia ChristensenPia Christensen

    Jennifer,
    Thanks for the heads up. The sentence is missing the “r” at the beginning of “rare.” It should read:
    “We conclude that despite widespread use of narcotic drugs in hospitals, the development of addiction is rare in medical patients with no history of addiction.”
    I’ll update the post right away.
    Best,
    Pia Christensen
    managing editor/AHCJ

Leave a Reply