Dan DiamondCMMI's ambitious goals
On its surface, Kolata's article is built around a reasonable conclusion: RCTs offer plenty of value in health care, and we'd benefit from more of them.
- As Jha alludes to, think of a double-blinded pharmaceutical study where half the participants randomly get a new drug and the other half get a placebo; that's an RCT.
- The famous RAND study that found having health insurance changes patients' behavior: An RCT.
- The ongoing Oregon Health Insurance Experiment: Also, an RCT.
And it's fair to examine how CMMI is pursuing its research, too.
Congress entrusted considerable authority to CMS's innovation center, which was created under the Affordable Care Act and designed to fast-track ideas and health reforms. If the center successfully uncovers a promising payment or delivery reform, an HHS secretary can scale it up nationally without having to wait on an act of Congress.
For now, CMMI has focused on launching about three dozen demonstration projects, like its Pioneer Accountable Care Organization program. A demonstration project is a better way to see how an intervention would fare in the real world, center director Patrick Conway told the Times. It also allows the center to make real-time changes, Conway pointed out. And "if it does not look like [a project] is working, we can stop.”
What's the state of ACA implementation? Join us for a conversation on March 19 as our experts perform a pulse check on Obamacare.
Would RCTs even work for CMMI?
CMMI's approach isn't totally above reproach; the data that the center is seeing from its pilots could be confused by secular trends, like changes in population, practices, and so on. That's why, Harvard's Jha acknowledged, it's important to design studies with a contemporary control group and statistical testing.
But under CMMI's ambitious charter, researchers are attempting to track a range of payment and delivery reforms. And it's hard to think of how the center could use an RCT for some of its projects.
For example, I asked a half-dozen different researchers to construct a hypothetical RCT to test how accountable care organizations would work. All were stumped.
"You'd have three patients, focus on one intervention, and track them for six months," one joked to me.
What randomized controlled tests can (and can't) offer
Researchers are careful to stress that RCTs can be very valuable in health policy. The reaction to new data coming out of the Oregon Health Insurance Experiment is an example of that; the OHIE has been closely scrutinized because there's so much interest in figuring out how the Affordable Care Act's coverage expansion will affect patient behavior and health.
But the effort to conduct an RCT can be prohibitive. If RAND attempted its seminal experiment today, it would cost half a billion dollars, says Adrianna McIntyre, managing editor of The Incidental Economist blog.
And CMMI's just not equipped for that kind of work, McIntyre added; its $10 billion, 10-year budget may seem huge, but that's still just 0.1% of Medicare and Medicaid spending over the next decade.
"RCTs attempt to control for all unplanned variables to the greatest extent possible," Su points out, "and that is nearly impossible to do when you consider the scale and diversity of models that CMMI is pursuing…multi-year, multi-stage models that may not lend themselves to a traditional RCT timeline."