Making sense of hospital quality reports

By Phil Galewitz

This spring marks 20 years since a tiny state agency called the Pennsylvania Health Care Cost Containment Council became the first in the nation to report hospital mortality data that was intended for release to consumers. I was there when the report was released to much excitement from health reporters in Harrisburg and to a nervous hospital industry worried how the public would perceive the data.

Back then hospitals said the data would be misleading and only confuse consumers. Today, many states – along with the federal government and numerous private organizations – regularly publish or put online report cards on hospitals. But it is uncertain if consumers are paying attention. And journalists are often unclear about the potential biases and limitations the reports present.

A report in the December issue of Health Affairs found serious flaws in the public reporting of hospital quality data. In this Q&A, one of the authors, Michael Rothberg, M.D., M.P.H., a researcher at Tufts University in Boston, discusses the findings. AHCJ also offers a number of resources for reporters who are looking at hospital quality.

Q: Health journalists often publish the results of state and federal quality data to their readers. Your study suggests that this might be a disservice. Is that right? 

Rothberg: Helping readers to focus on the quality of their health care is a great idea in theory, but in practice we have very crude tools for measuring quality, and our study suggests that consumers (or reporters) who follow only one rating service will not be seeing the whole picture.  If that limited view keeps a patient from going to a hospital that they prefer for other reasons, then it could be seen as a disservice.

Q: Do you think the publicly reported data should drive patient decisions on where to seek care or at least spur them to ask questions of their doctor about which hospitals to use? 

Rothberg: For some problems, such as coronary artery bypass surgery, where the patient has time to do research, the process is analogous to a production line, and hospitals have sufficient volume to smooth out chance events, it would make sense to rely on quality metrics.  Unfortunately, in our study we found that even for bypass the results of the different rating services were conflicting.  Unfortunately, I don’t think doctors are much savvier about quality measures than educated patients.

Q: The lack of standardized risk-adjusted methods has been a problem for decades in studying outcomes data. Why haven't researchers been able to agree on how to risk adjust?  Isn't it time the federal government intervened on this issue? 

Rothberg:  Risk adjustment has been a problem for over a hundred years, since Florence Nightingale first noted that hospitals had vastly different mortality rates.  The problems have to do with disparate models—there is no official arbiter to decide the “right” model to use—and also with the quality of the data that go into the models.  The government could choose a best model for each disease state and also mandate that hospitals provide all the patient information required to calculate each hospital’s adjusted rate, but this would cost a lot of money, which would invariably come from our healthcare dollars.  That is, hospitals would have to spend money documenting the care provided, instead of spending that money on actually providing healthcare.  Unless we knew that the effort would actually lead to improvements in quality, I’m not sure we could afford it right now.

Q: Do you think consumers trust certain rating services more than others? For example, might consumers be more likely to look at state or federal government hospital quality reports than a private company such as HealthGrades? 

Rothberg: Some people trust the government, some don’t.  The fact is that HealthGrades gets all its information from Medicare anyway — it is simply repackaging government-collected data.  I think that the government is trying very hard to provide the best possible data to consumers and to get them to use it.  The problem is that for the most part we just don’t have great quality measures.

Q: Though consumers may not spend much time looking at hospital quality data, what about large employers and health insurers? Are they using the data to determine where to steer their patients or employees? 

Rothberg: Large employers are interested in improving the quality of care their workers receive, which is how Leapfrog started.  Insurers also want to provide high quality care.  Some insurers have experimented with discounts to patients who seek high quality care (or penalize patients who go to “bad” doctors, as if that weren’t punishment enough).  For the most part, however, contracts between insurers and providers are still governed by price, with discounts for volume, not quality.

Q: What hospital data is most important to consumers? Is it mortality data? Volume data? Patient satisfaction data? Or do consumers need it all even though the data put together might be confusing such as a hospital having low mortality for a service, yet low patient satisfaction and low volume. 

Rothberg: There is actually very little research on what consumers want in this area.  The few studies find that consumers are less interested in confusing quantitative data like whether patients had blood cultures drawn before antibiotics were started, and more interested in the experiences of other patients.  They want to know whether their pain will be addressed quickly and completely, whether the doctor will listen to them and explain things in language they can understand, and whether a nurse will come when they call.

Q: Hospitals pay HealthGrades if they want to use the rating service's rankings in any marketing or promotions. Is than an inherent conflict of interest that consumers should be leery about when they see a hospital touting HealthGrades results? 

Rothberg: Hospitals pay to participate in various rating services and then tout their rankings on their website, etc.  I suppose there is some conflict of interest in any private rating agency, but I’m not sure we’ve gotten to the level of acceptance where these relationships make much difference.  The stakes just aren’t that high.

Q: A number of states have started publishing hospital infection data showing which hospitals have higher or lower than average infection rates. Is this a useful endeavor because consumers will look at the data or because it may scare hospitals to clean up their act? 

Rothberg: More the latter.  Hospital management is very aware of this sort of reporting, and no one wants to be the CEO of the most dangerous hospital in California.  The trick here is making sure that the infection rates are accurate, and that hospitals are only cited for being statistically better or worse than average.

Q: How can journalists use the publicly reported data on hospitals to best help readers and serve as a watchdog on their local hospitals? 

Rothberg: Although patients are not particularly interested in these ratings, hospital administrators are acutely sensitive to them.  By reporting hospitals that appear to have a quality problem, reporters can very easily get the attention of top management, who will make changes to get out of that uncomfortable spotlight.

Q: Any other advice you would give to health reporters? 

Rothberg: One other thing we found was that quality in one area (e.g. cardiac surgery) did not necessarily translate into quality in another area, such as hip replacement surgery.  So it is not possible to simply identify a “best hospital” and go there for everything. Unfortunately, for many procedures, hospitals simply don’t have enough volume to know whether the results reported are due to chance.  Reporters should remain sensitive to the fact that hospitals provide many different services under a single brand.  Some of these services are better than others, but the data don’t always tell us which is which.

---

Michael Rothberg (Michael.Rothberg@bhs.org) is an assistant professor of medicine at Tufts University School of Medicine, Baystate Medical Center, in Springfield, Mass. Rothberg is the recipient of a Doris Duke Clinical Scientist Development Award.

Phil Galewitz is a health writer for The Palm Beach Post, an AHCJ board member and editor of HealthBeat.