CNN in the hot seat: Using numbers without context has consequences

Photo: stevenharman via Flickr

How many times have you wanted to make a comparison between two numbers — a local rate and national rate, or some kind of rate for one type of surgery vs another, or one demographic group vs another … but you didn’t have the comparison statistics you needed?

What did you do? Did you write around the issue and choose a different angle or framing? Did you cobble together the number you needed from different sources? Did you use a similar number but include qualifications about limitations of the comparison?

The most recent appeals court decision in a libel case against CNN reveals why these questions matter — a lot. While the final outcome has yet to be heard and decided, the network’s failure to ward off the suit altogether provides a cautionary tale as much as it reveals a familiar frustration for reporters in search of transparency.

It’s tough to summarize the whole situation without wading through Amazonian weeds, but here are the basics any health reporter will at least want to be aware of:

  • In June 2015, CNN ran a comprehensive, multimedia investigative story about the pediatric cardiac surgery program at St. Mary’s Medical Center in West Palm Beach, Fla.
  • CNN alleged St. Mary’s had a pediatric open-heart surgery mortality rate of 12.5 percent, a rate that CNN alleged was three times the national average of 3.3 percent.
  • The 12.5 percent was a calculation CNN made based on data they pulled together from various sources because St. Mary’s would not publish or otherwise provide their rate. Trudy Lieberman has a detailed breakdown on what CNN did in a July 2015 CJR article about the case; her article provides an excellent overview of the whole issue.
  • The 3.3 percent apparently actually referred to the national mortality rate for all thoracic surgeries in the Society for Thoracic Surgeons database. (A primer on the strengths, limitations and considerations of using the Society of Thoracic Surgeons (STS) National Database in research was just added to the Medical Studies Data section.)
  • CNN did not make a risk adjustment for their 12.5 percent figure. Though raw figures of mortality rates and similar figures are not worthless, they definitely lack important context about demographics, geography, payers and potentially relevant hospital characteristics that can affect raw mortality rates.
  • The Society of Thoracic Surgeons does use a risk adjustment in comparing surgery-related mortality rates across institutions.
  • Mary’s alleges that their risk-adjusted mortality rate for congenital heart surgery patients at the time of CNN’s reporting was 5.3 percent — not statistically different from the national average. (They later published an updated figure of 4.7 percent for the present risk-adjusted rate.)
  • Davide Carbone, former CEO of St. Mary’s, says he was forced to resign as a result of the CNN story, and the hospital’s pediatric cardiology program has ended.
  • Carbone sued CNN for libel because of what he alleges were misleading mortality rates and false reporting.
  • CNN attempted to use a common legal counterstrike in libel cases, called an anti-SLAPP (Strategic Lawsuit Against Public Participation) statute, to get the case dismissed.
  • A federal district court found in favor of Carbone — that the lawsuit could move forward — in early 2017. CNN appealed.
  • The Eleventh Circuit Court of Appeals just upheld the district court’s ruling, agreeing that Carbone has a case and the case can move forward.

So, what does all this mean and why should you care?

CNN has not been found guilty of libel or false reporting. The two rulings were on the validity of the case itself — whether Carbone has a right to move forward with his case and have it heard in court. This is still a substantial setback for CNN because it will now have to pour a fair amount of time, energy and legal fees into a full-fledged defense at trial or else settle with Carbone. Either option is costly.

I’m not a judge, a thoracic surgeon, a hospital CEO, a biostatistician or a reporter/producer on the story, so I cannot say how egregious CNN’s actions were and whether they rise to the threshold of false or misleading reporting. But, CNN certainly put itself on thin ice by trying to come up with their own numbers and — most importantly — not providing context for their numbers.

Sure, St. Mary’s should have been transparent — lack of transparency is one of the greatest sins and frustrations of the U.S. health care system — but an uncooperative source is not an excuse for inadequate context. CNN should have explained exactly how it arrived at 12.5 percent, noted that it was not risk-adjusted, explained what risk-adjustment meant, and probably brought in a biostatistician for an assessment of their calculations.

This matters because health reporters every day struggle to find the precise statistics they need for a story, and CNN’s experience shows how disastrous cutting corners can be. Was CNN’s reporting so misleading and clumsy as to be construed as negligent libel? I don’t know. But the network could have avoided this whole situation with more diligence and attention to context in the first place.

3 thoughts on “CNN in the hot seat: Using numbers without context has consequences

  1. Avatar photoPaul Burke

    Most risk adjustment is pitifully poor, and does not necessarily deserve to be included in a general news story. Any mention or use of risk adjustment needs to reveal how poor it is. Since risk adjustment is ineffective, hospitals can improve their results by denying care to the patients with the worst conditions (“We can’t help you…”), giving them a better “success” rate. So I agree with you the qualitative measures are the most important. Outcome measures lead to denial of care. Not only will the libel suit chill reporting, but in the opposite direction, reporting which focuses on outcome measures leads to denial of care to the sickest.

    “the most assiduous work on risk adjustment has produced tools of only moderate power. The prospects for solving this problem with improved risk adjustment are not promising.[4],[5]”

    “You can’t get all the right variables on the page” said Berwick, former Medicare Administrator.

    The Society of Thoracic Surgeons (STS) does not seem to publish the reliability of their adjustment for congenital heart surgery. mentions congenital heart surgery without measurement measures reliability without congenital heart surgery measures reliability without congenital heart surgery

    For other procedures, STS gives “c-statistics” ranging from 0.616 to 0.826. The c-statistic has a scale of 0.5 to 1, where 0.5 means their equations do no better than chance, and 1 means their equations are perfect. So some equations are little better than chance, and they still rate hospitals with them. “Models are typically considered reasonable when the C-statistic is higher than 0.7 and strong when C exceeds 0.8”

    As a much more professional and problematic example, Medicare’s adjustment of health condition (HCC) is equally poor. They explain only 2% to 12% of the total variation actually caused by patient mix (p.65 table 3-22, “r-squared” of version 21).

    Medicare’s adjustment of patient mix for readmission penalties is also poor. For example, their equations explain 3% of the variation in readmissions among heart failures (p.30), 5% for heart attacks (p.30) and pneumonia (p.29). These percents date from 2008 and have not been updated.

    Medicare now shows c-statistics between 0.61 and 0.66 for readmission penalties, so none of their equations is “reasonable,” and they still charge hospitals hundreds of millions of dollars of penalties with them each year, driving hospitals to reduce admissions among the sickest.

  2. Avatar photoMichael Millenson

    Paul: This is an impressive reply. For those of us unfamiliar with you and your work, might you provide a bit more background? My Googling gives me a guess, but I am not confident with a statistically reliable interval that I’m right. Regards, Michael

  3. Avatar photoPaul Burke

    Michael: This “About” page on my site gives some background:

    I have a numbers background, but I’m not a biostatistician, and I agree with Tara’s advice that reporters gain by talking to “a biostatistician for an assessment of their calculations.” Major institutions like Medicare and hospitals can quote a lot of statistics, which may not always be significant.

Leave a Reply