Surgeons’ complication rates become public with new database

Whether consumers are choosing a car, a household appliance or even a nursing home, there are ratings and reviews available to make the best choice. But patients are often blind when choosing a surgeon.

Surgeon Scorecard, a database released by ProPublica this week helps shed some light on that area with an analysis of death and complication rates for nearly 17,000 U.S. surgeons for eight common surgical procedures. This is the first time this information has been available to the public.

Sisi WeiOlga Pierce and Marshall Allen used Medicare billing records for in-patient hospital stays from 2009 through 2013. They excluded cases that involved trauma and high-risk cases and adjusted the analysis to account for patients’ health and age, luck and the hospital’s overall performance. They involved panels of doctors in the analysis, including surgeons who specialize in the rated procedures. Read more about the methodology here.

As Pierce and Allen point out, most patients think it’s important to choose a good hospital. But they found “that even within ‘good’ hospitals, performance between surgeons can vary significantly. Half of all hospitals in America have surgeons with low and high complication rates.”

While the Centers for Medicare & Medicaid releases data about hospitals’ overall performance through Hospital Compare (data also available from AHCJ), the agency hasn’t required hospitals to track death rates and complications by doctor. According to Pierce and Allen, “Some experts say it’s time to put more emphasis on the role of individual practitioners.”

“The idea that it’s all systems and there are no individual performance differences is absurd,” said Dr. Robert Wachter, chief of medical service at the University of California, San Francisco Medical Center and a nationally known expert on patient safety. “A good system has a mechanism to identify poor performers and either make them better or get rid of them.”

ProPublica acknowledges that plans to publish the Surgeon Scorecard were controversial, even noting that Johns Hopkins physician and patient safety expert Peter Pronovost, M.D., first lauded the project but more recently has criticized the effort. That criticism is included on a page intended to further the conversation about patient safety that invites experts to weigh in on the project.

A number of Gannett news organizations are using the data to report on local doctors; see a collection of the stories they’ve found.

4 thoughts on “Surgeons’ complication rates become public with new database

  1. Avatar photoNorman Bauman

    Patient income?

    Interesting. I’ve written about surgical outcomes, and read some of the literature, most recently total knee replacement. I will read this with great interest. But it’s so difficult to correct all the patient factors in database studies.

    Here’s an article on socioeconomic factors:

    Impact of Socioeconomic Factors on Outcome of Total Knee Arthroplasty
    Robert L. Barrack, Erin L. Ruh, Jiajing Chen et al.
    ClinicalOrthopaedics and Related Research
    DOI 10.1007/s11999-013-3002-y

    Barrak et al. surveyed 661 patients 18-60 years (not Medicare), 1-4 years after TKA at 5 orthopedic centers, funded by Stryker.

    Patients reporting incomes of less than $25,000 were less likely to be satisfied with TKA outcomes and more likely to have functional limitations than patients with higher income.

    Less than $25,000: 18% dissatisfied.

    Equal to or greater than $25,000: 8% dissatisfied.

    That makes sense. In Jane Brody’s NYT story about her own TKR, she said that her insurance company wouldn’t pay for the recommended number of physical rehabilitation sesions, so she had to pay for more sessions herself. In contrast, someone in AHCJ found a woman on Medicaid in California who had received a TKR, but couldn’t get any physical rehabilitation.

    As far as I could tell, the Surgeon Scorecard didn’t consider patient income. I don’t think the Medicare database has patient income. It might be that some surgeons and hospitals have wealthier patients than others, and they get better outcomes simply because of socioeconomic factors. You might be measuring the patients’ wealth, rather than the surgeon’s skill.

    The overall outcomes in Barrack et al. aren’t that great. 27% of men and 36% of women had pain at least 1 year after surgery. 26% of men and 33% of women had problems getting out of a chair. They’re not dancing like Jane Brody.

  2. Avatar photoBen Harder

    Norman, you’re correct that the Medicare database does not contain data on patient income, which makes it very challenging for health services researchers and journalists to adjust for socioeconomic status (SES). Nevertheless, experts are increasingly recognizing that it’s important to adjust for SES when drawing comparisons among healthcare providers, whether hospitals, surgeons or other providers.

    In our (U.S. News & World Report) recent investigation of hospital outcomes in THR, TKR and three other common procedures and conditions, we made an adjustment for SES by including as a variable in our risk model whether or not each Medicare patient we analyzed was dual eligible for Medicaid. We feel that adjustment is a step in the right direction, but it’s by no means a complete adjustment for SES. More data needs to be made available to help journalistic organizations like U.S. News and ProPublica make the most rigorous possible comparisons among providers.

  3. Avatar photoNorman Bauman


    After I wrote that, I read the methodology paper

    2.5 Patient health

    A notable feature of our model is that the small but significant effect of the Health Score (per-procedure AUC of .57-.63) essentially disappears when age and the hospital and surgeon effects are included in the model. This shows that the quality of care is likely a more important factor determining patient outcomes.

    2.6 Income Sensitivity analysis

    A sensitivity analysis was conducted using each hospital’s SSI rate as a proxy for patient socioeconomic status. The rate expresses what portion of a hospital’s care to Medicare patients goes to patients who qualify for Supplemental Security Income, a federal welfare program which requires very low income and assets for eligibility.

    SSIrate = MedicareSSIdays/TotalMedicaredays

    Including the additional variable had little effect, perhaps because the metric refers to a hospital’s patients as a whole, not the specific group of patients undergoing elective surgical procedures. It it also possible that some of the effect often attributed to patient socioeconomic status is actually due to the effect of the hospitals and surgeons who provide them care.

    You say that it goes against the conventional wisdom, and I agree with you on that. In the stories I wrote (focusing on abdominal aortic aneurysms, but also on whatever procedures had been best studied, such as colon cancer), I found that there were variations in hospitals and surgeons, but the hospital was more important. The best proxy I found was surgical volume.
    What Makes A Good Surgeon? What Makes A Good Hospital?
    September 10, 2007
    By: Norman Bauman for Body1
    Dr. Peter RF Bell: Innovation in Vascular Surgery
    April 10, 2007
    By: Norman Bauman for Veins1

    You also seem to be saying that patient health cancels out after you account for age. I talked to surgeons who do these procedures, and that’s not what they say. Peter Bell told me that if he wanted good numbers, “I could just do easy cases.”

    In AAA surgery, the two factors that affected mortality were lung and kidney capacity.

    If you look at the curves of lung function vs. age for different populations, the lung function of smokers declines with age a lot faster than non-smokers. A 72-year-old non-smoker might be able to walk up 10 flights of stairs, while a 55-year old ex-smoker with COPD might not have enough lung function for walking. That’s why anesthesiologists have pre-surgical consultations.

    Similarly, the kidney function of people with diabetes or hypertension declines with age a lot faster than healthier people.

    In 2.6 you say that the percentage of hospital SSI had no significant effect on outcomes, and therefore you assume that socioeconomic status had no effect on individual surgeon’s performance. (This is an important issue because HHS plans to penalize hospitals on outcomes, without correcting for socioeconomic status.) I’d like to see better evidence on this point.

    I think of Jane Brody getting total knee replacements, and paying for additional post-surgery physical therapy sessions out of pocket. Then I think of a Medicaid patient getting total knee replacements, and not getting post-surgery physical therapy sessions at all. How can they have the same outcomes?

    Surgeons always say, these debates are good, because they benefit patients. So it’s good to have studies like yours, but you have to challenge them too.

    BTW, UK hospitals have been posting outcomes by individual surgeons for years. They decided that taxpayers pay for the NHS, so that data belongs to the taxpayers.

  4. Avatar photoBen Harder

    Norman, I absolutely agree with you that socioeconomic status has a potentially important effect on outcomes. That’s why groups like U.S. News, ProPublica and academic researchers conduct tests like sensitivity analyses to gauge the impact of SES on the outcomes we’re measuring.

    I do want to point out that in your reply you appear to be giving me credit for work ProPublica has done. Marshall Allen, Olga Pierce and others put in an enormous amount of effort into their very important analysis, and I want to make sure they – not I – get the credit. While my team at U.S. News has used methods that are in some ways similar to theirs, I played no role in their project.

Leave a Reply