CMS has been miscalculating hospitals' overall star ratings for about two years, according to an analysis conducted by Rush University Medical Center in Chicago, Maria Castellucci reports for Modern Healthcare.
Just updated: Get the cheat sheet on how CMS calculates their star ratings
According to Modern Healthcare, the findings—which may explain why CMS earlier this week announced it would delay its scheduled July update—could have significant implications for hospitals, as consumers commonly use star ratings to determine where to get care, and payers often use the ratings in contract negotiations.
CMS' Hospital Compare website's Overall Hospital Quality Star Ratings rate more than 4,000 U.S. hospitals on scale of one to five, with five stars being the highest. Hospitals can report data on up to 57 measures across seven categories:
- Effectiveness of care, weighted at 4%;
- Efficient use of medical imaging, weighted at 4%;
- Mortality, weighted at 22%;
- Patient experience, weighted at 22%;
- Readmissions, weighted at 22%;
- Safety of care, weighted at 22%; and
- Timeliness of care, weighted at 4%.
Hospitals must report measures for at least three of the above categories, including at least one outcome measure:
- Readmissions; or
Each category contains specific metrics that Modern Healthcare reports "are supposed to be evenly weighted to calculate the hospital's performance in that area." But Rush's analysis, which the medical center shared exclusively with Modern Healthcare, suggests CMS is not evenly weighing measures in the safety category.
Rush officials first raised concerns about CMS' methodology in May, when the medical center learned its star rating would fall from 5 stars to 3 stars when CMS published the now-delayed July update. (CMS discloses star ratings to hospitals two months before they are published to give hospitals time to review the ratings.) Omar Lateef, an author of the analysis and Rush's senior vice president and chief medical officer, said CMS initially dismissed the concerns, prompting the medical center to conduct its own analysis.
For the analysis, Rush compared the methodologies CMS' used to calculate the now-delayed July update and the previous release in December 2017 and compared the results. According to Modern Healthcare, leaders at the Association of American Medical Colleges, UChicago Medicine, and University of Virginia Health System contributed to the analysis.
They found that December's star ratings update had heavily weighted one safety of care measure—PSI-90, which measures the frequency of safety issues, such as pressure ulcers and sepsis rates—as opposed to weighing all measures within that group evenly. For instance, the analysis found the PSI-90 weight was:
- 1,010 times higher than the catheter-associated urinary tract infections measure;
- 81 times higher than the C. difficile infection rates measure;
- 51 times higher than the central line-associated bloodstream infection rates measure; and
- 20 times stronger than the surgical site infection rate measure.
According to Modern Healthcare, the PSI-90 metric previously "presented the most variation in performance between hospitals so it [was] weighted heaviest."
But for the now-delayed July ratings, Rush found CMS' formula placed a greater emphasis on complication rates from hip and knee replacements. According to the analysis, that metric alone accounted for about 98% of a hospital's performance in the safety category.
Rush also concluded that a hospital's overall star rating can be greatly influenced by the safety of care group, so changes in the way that score is calculated could—and in Rush's case did—greatly affect a hospital's overall star rating, Modern Healthcare reports.
Lateef said, "Given the disproportionate weighting of the safety scores over time, they did not represent a composite measure."
According to Modern Healthcare, CMS told Rush officials it had changed the way it traditionally calculates the PSI-90 measure for the now-delayed July ratings so that it relies on new ICD-10 codes. CMS said that changed the way the agency's model calculated the safety category, ultimately placing a greater emphasis on the hip and knee complication metric.
While Modern Healthcare reports academic medical centers, like Rush, for years have opposed CMS' star ratings methodology, other experts also have pushed back against CMS' formula.
David Levine, senior vice president of advanced analytics and informatics at Vizient, said CMS relies on latent variable modeling to calculate star ratings, which Levine said is not appropriate for measuring clinical outcomes. "We have expressed our deep concerns about this methodology because it changes the weight every time—that doesn't really make sense," he said
According to Modern Healthcare, the latest findings could explain why CMS delayed the July update. When asked to comment on Rush's analysis, CMS referred to the announcement it made earlier this week explaining the delay was "to address stakeholders concerns," Modern Healthcare reports (Castellucci, Modern Healthcare, 6/15).
Just updated: Learn how CMS Star Ratings—and 6 other ratings programs—actually work
Download our one page sheets for summaries on the methodology and metric categories used in seven hospital quality rating programs:
Get all the Cheat Sheets
Next in the Daily Briefing
Are hospitals actually cutting readmissions—or is that just fuzzy math? A new study deepens the controversy.