It’s inevitable that when any organization publishes a report card on physician quality, doctors and policy experts will critique its methodology. Therefore, ProPublica surely is not surprised see criticism of the extensive work that Sisi Wei, Olga Pierce and Marshall Allen did sorting through Medicare billing records for 17,000 U.S. surgeons at 3,575 hospitals who performed eight types of surgery from 2009 through 2013. The analysis formed the database of the nonprofit newsroom’s Surgeon Scorecard launched in July. (We covered the scorecard here: Surgeons’ complication rates become public with new database.)
The latest criticism comes from the Rand Corp. In a 19-page report issued today, A Methodological Critique of the ProPublica Surgeon Scorecard, nine Rand researchers say their critique had three goals: to explain issues they have with the scorecard’s methodology; to suggest ways to improve the scorecard; and to inform the public about these issues. By informing the public, the researchers said in their Sept. 25 critique, “…we hope patients who are choosing a surgeon will be better able to decide how much weight to give the data presented in the scorecard.”
In producing an online, searchable database for the public, ProPublica’s goals are laudable, the researchers said, but the scorecard’s ability to achieve its goals is limited by the rigor of the methods and the adequacy of the underlying data. They point out five specific issues they have with the methodology, and outline six ways to improve the scorecard.
Until these flaws are addressed, the Rand researchers suggest that patients should not consider the scorecard to be, “a valid or reliable predictor of the health outcomes any individual surgeon is likely to provide.”
Stephen Engelberg, ProPublica’s editor-in-chief, responded via email today. “The issues Rand raised have been previously addressed in our editor’s notes here and here and in our FAQs, and a white paper,” he said. “In the white paper, you’ll see the names of several colleagues of the Rand authors who helped us develop our methodology.”
“As the Rand paper shows, reasonable scholars can and do disagree about how to calculate complications,” Engelberg continued, pointing to ProPublica’s “What Experts Are Saying” page about the project.
“We appreciate Rand’s input as we develop Surgeon Scorecard 2.0 and continue to believe that our approach provides patients with valuable information that was not previously available,” he said, adding that ProPublica hopes to have the update ready by the end of the year.
[Editor’s note: The headline of this post has been changed to more accurately reflect the Rand report and the spelling of Stephen Engelberg’s name has been corrected. We apologize for the errors.]
