Medicare's star ratings are biased, new analysis argues

CMS' overall hospital star ratings released last December disproportionately favored specialty hospitals due to the agency's method for rating institutions that didn't report data on all 57 measures, according to a Modern Healthcare analysis.

Map: New overall star ratings are out. How did your hospital fare?

How the ratings work

CMS' Overall Hospital Quality Star Ratings on its Hospital Compare website rates more than 4,000 U.S. hospitals on scale of one to five, with five stars being the highest. Hospitals can report data on up to 57 measures across seven categories:

  • Effectiveness of care, weighted at 4%;
  • Efficient use of medical imaging, weighted at 4%;
  • Mortality, weighted at 22%;
  • Patient experience, weighted at 22%;
  • Readmissions, weighted at 22%.
  • Safety of care, weighted at 22%; and
  • Timeliness of care, weighted at 4%.

Hospitals must report measures for at least three categories, including at least one outcome measure: readmissions, safety, or mortality. CMS weighs the data submitted more heavily when a hospital does not report for all measures, according to Modern Healthcare.

Critics have argued that the weighting process allows some hospitals to appear to perform better or worse by submitting more or fewer data. Nancy Foster, VP for quality and patient safety at the American Hospital Association (AHA), said, "You are getting this blend of measures that aren't adequately adjusted … you end up with this strange distortion."

What Modern Healthcare found

For the analysis, Modern Healthcare worked with consulting firm Sullivan, Cotter and Associates to review the latest star ratings. Sullivan Cotter used its own methodology to identify major teaching and specialty hospitals, Modern Healthcare reports.

The analysis found that specialty hospitals were more likely to receive five and four stars under CMS' new ratings system than major teaching hospitals. Of the 74 specialty hospitals that received a star rating in December 2017, 61% received five stars and 22% received four stars. By contrast, of the 172 major teaching hospitals that received a star rating in December 2017, 9% received five starts and 15% received four stars.

Further, the analysis showed that specialty hospitals reported an average of 27.2 of the 57 measures, while major teaching hospitals reported an average of 51.37. According to the analysis, specialty hospitals were less likely to report data on mortality than the 172 major teaching hospitals, which reported all measures needed to meet the patient experience, readmissions, mortality, and safety categories.

Jeff Softcheck, principal at Sullivan Cotter, noted that the specialty hospitals report fewer measures because their patients do not present with relevant conditions. For example, it would be unusual for a specialty hospital to provide care for congestive heart failure or stroke patients.

Softcheck further explained that CMS' methodology is set up in a way that means hospitals that perform below average on any of the four high-weight measures have to perform above average on the three remaining measures to receive at least four stars.

According to the analysis, just over half of major teaching hospitals that received four or five stars performed below average on readmissions. To recoup, most performed above average on mortality, patient experience, and safety.

Discussion

CMS has defended its methodology, which was updated December 2017. A spokesperson said the methodology "is not designed to advantage or disadvantage any type of hospitals."

But AHA has suggested that CMS perform better risk adjustment so that hospitals are compared to others with similar patient populations and service offerings.

Other experts have said CMS should measure and publish ratings based on performance on specific conditions. David Nerenz, the director of the Center for Health Policy and Health Services Research at the Henry Ford Health System, said the star ratings don't show whether a hospital is a good choice for a certain procedure because it combines all "these measures that aren't correlated with each other—that is the fundamental problem" (Castellucci, Modern Healthcare, 3/14; Zimmerman, Becker's Hospital Review, 12/22).

Get 1-page cheat sheets on other Hospital Quality Rating Programs

Download our one page sheets for summaries on the methodology and metric categories used in five hospital quality rating programs:

Get all the Cheat Sheets


Next in the Daily Briefing

Weekend reads: Turns out, you can catch the same cold twice

Read now