Nursing home star ratings are misleading and disingenuous, according to a recent analysis comparing ratings with quality measures alone. More than a thousand nursing homes nationally with high overall ratings had only one or two stars in quality measures, which could point to some serious health implications for residents.
Quality measures (QM) is one of three categories in the star system used by the Centers for Medicare & Medicaid Services. Each nursing home receives one overall five-star rating and separate ratings in health inspections and staffing, and for QM. The QMs offer some insight on how well these facilities are caring for their residents’ physical and clinical needs.
These ratings provide information on 11 physical and clinical measures, including use of antipsychotic medications for both long-stay and short-stay residents, according to the CMS website. (CMS recently updated its criteria to include 13 out of 24 measures).
San Diego content and data visualization company 1point21 Interactive conducted an initial review of star ratings in Pennsylvania for the Philadelphia law firm Rosenbaum and Associates. They found that 49 nursing homes in Pennsylvania — nearly 20 percent of long-term or skilled facilities with a four- or five-star overall rating — had QM ratings of only one or two. Other states with high incidence of discrepancies between overall and quality ratings were Texas, Ohio, Illinois, Missouri and Florida.
“Because of the way CMS awards and subtracts points for each category, it’s possible to score a one or two on QM and still end up with a four or five overall rating,” Brian Beltz, 1point21’s content manager and research coordinator, said in a phone interview. High scores in the inspection and staffing categories, for example, can bump up a facility’s rating despite a low QM score.
“Unfortunately, many consumers don’t know this and think that if a nursing home gets a high rating, they’re strong on all measures,” Beltz said.
The process “is so confusing there is no way the average consumer can figure this all out,” Toby Edelman, senior policy attorney at the Center for Medicare Advocacy, told me by phone.
Since most of the metrics are self-reported, “quality measures are not very reliable to begin with,” Edelman said. Facilities essentially “work to the test,” focusing on the metrics that CMS more strongly considers in the ratings.
CMS publishes tech guides to help users navigate and understand the rating system. Unfortunately, these can be so dense and filled with jargon that even experts like Edelman admit they have difficulty making sense of them.
How they did it
The idea to look at quality measures came from ProPublica’s Nursing Home Inspect. Beltz said he couldn’t find anything comparable online so decided to tackle the project himself. He thought he might find a few trouble spots, but never expected to find 1,127 homes across the country still considered “good” or “very good” overall despite low QM ratings.
Beltz used CMS’ May 1 Nursing Home Compare Star Rating and minimum data set (MDS) quality measures databases. In addition to the overall QM rating, he looked at seven QM metrics specific to long-term care:
- Pressure ulcers.
- Losing too much weight.
- Catheter inserted and left in bladder.
- Urinary tract infections.
- Antipsychotic information.
- Ability to move worsened, anti-anxiety or hypnotic medication.
The facility rating for the QM domain is based on performance in a subset of the MDS-based quality measures and three MDS- and Medicare claims-based measures posted on Nursing Home Compare, according to CMS.(Five additional long and short-stay measures were added, effective July 1). The specific metrics Beltz analyzed could have many serious health implications for patients, he said.
It took a few days to scrub and clean the datasets – just the quality measures data alone included some 676,000 records. Beltz’s team then culled the Pennsylvania-specific data, which was tabulated and mapped. Potentially troublesome facilities throughout the state were highlighted.
“We tried to get some answers from the state nursing home ombudsman and other state officials and received back form letters saying essentially that they were doing their best,” he said.
Jeff Porter, AHCJ’s special projects director, spot-checked the Pennsylvania data, and found the conclusions accurate.
A caveat about star ratings
Edelman said it’s hard to trust the ratings when five-star facilities get sued for wrongful death or a shift in management can change things for better or worse. “Don’t rely on it,” she advised. This New York Times article looked at how some nursing homes can game the system.
Consumers are often at a complete loss when trying to choose a facility, especially since they often are under immense stress and time pressure. According to Edelman, the most important aspect of star ratings are actually the staffing ratios. “Adequate numbers of nurses and aides will do a lot to improve poor quality measures,” she said.
Well before stars were part of CMS’ nursing home performance, this article in BMC Health Research, co-authored by AHCJ member Trudy Lieberman, noted:
“The authors doubt the ability of many nursing home performance systems to truly help consumers differentiate among homes providing different levels of quality. We also argue that, for consumers, performance measurement models are better at identifying problem facilities than potentially good homes.”
Edelman said that when it comes to star ratings, even facilities with a four or five do not paint a complete picture of care. A 2015 report from Abt Associates, which looked at six-year trends, found no change in distribution of star ratings since 2009. The report also found that after CMS recalibrated how it calculated quality measures, there was a “dramatic” change from 2009-15 in facilities receiving five-star QM ratings and about half as many one-star QM ratings.
Edelman calls QM ratings “ridiculous” and “unreliable,” but Beltz said he hopes the tool he created can help consumers consider more than just a facility’s overall star rating when searching for a nursing home. “There is a need and it should be easy for people to see and compare information,” he said.
While this tool this may be useful for some consumers with time to do the research, Edelman said the reality is that many family members get only a few hours’ notice when a loved one is about to be discharged from a hospital. It’s logical to eliminate any one-star facility, but more difficult to determine why some homes get four stars and others, five. “Nursing Home Compare is just a beginning tool, and under cites deficiencies,” she said.
There is much more to a rating than just stars, so she suggested reaching out to the state’s ombudsman, who will have the most current information on their state’s facilities and any significant deficiencies. Here are some tips for journalists covering this topic:
- Use 1point21’s searchable database to check out QM and star ratings for your state or community.
- Check quality measures along with metrics for other categories through Nursing Home Compare (as in this example) and compare scores to national and state averages. There’s a lot of information under each category. Scroll down for a look at all the measures.
- Remember that not all quality metrics go into the overall star rating for that category. But they may help you pinpoint potential trouble spots. For example, if pressure ulcers are a problem is it because of inadequate staffing or because of neglect?
- Many tools are available to help with data mapping. For this project, 1point21 used Carto.
