Not unexpectedly, ProPublica has published a response to the Rand Corporation’s recent 20-page critique of the journalism organization’s Surgeon Scorecard, a searchable database of complication rates for surgeons performing several elective operations.
In a Rand summary of the piece, its authors advised patients “not to consider the Scorecard a valid or reliable predictor of the health outcomes any individual surgeon is likely to provide.” The paper listed what researchers labeled “Methodological Issues” in the Scorecard and ways the authors thought the data could be improved.
ProPublica’s rebuttal, written by editor-in-chief Stephen Engelberg and deputy data editor Olga Pierce, offers specific responses to Rand’s points and says the organization plans to take some suggestions into account as it prepares the next version of the scorecard. However, it says conversation about the database should be “grounded in fact and evidence.”
We do, however, disagree with much of what is argued in the RAND essay. As we will detail below, the authors omit mention of key aspects of our methodology and mischaracterize others. Several of the most significant purported shortcomings involve hypothetical events that “might be” true or are “likely.”
When the Scorecard was published, it was accompanied by a detailed explanation of ProPublica’s methodology, including some of the limitations of the data, as well as an article and editor’s note.
