The Blueprint

How can PCMH impact be "inconclusive"?

by Amanda Berra

A new literature review says that PCMH efficacy has yet to be fully demonstrated--at a moment when positive findings about PCMH impact have been rolling in.  Why the disconnect?

Background: PCMH as poster child for quality and cost improvements

In recent weeks, headlines have repeatedly celebrated the efficacy of the PCMH model. One study has just pegged Geisinger’s PCMH-related cost savings at 7.5% over four years. Insurer WellPoint is investing $1B in primary care based on its findings on downstream savings. Community Care of North Carolina (CCNC) reports saving nearly $1B over four years of PCMH.

Less than glowing headlines appear

With all this positive momentum backing the model, it made for a startling contrast when providers began seeing headlines like "Medical Home Cost Savings Questioned". 

This new set of articles referred to the fact that the Agency for Healthcare Research and Quality (AHRQ) commissioned a set of researchers from Mathematica Policy Research to review the evidence of the efficacy of the PCMH. The findings, released in the  the Feb. 2012 issue of the American Journal of Managed Care as "Early Evaluations of the Medical Home: Building on a Promising Start," by Deborah Peikes at al, indicates that after careful review, the team determined that evidence for medical home ROI is thin. 

Several participants of the Medical Home Project forwarded us links to the articles about the study, asking "Is PCMH efficacy really in doubt?"

The Study

 "Early Evaluations of the Medical Home: Building on a Promising Start,"  is obviously solidly researched, well written, and refers to a large base of information about PCMHs.  It contains two striking pieces of information:

1. The authors say that "the evidence indicates some favorable effects on all three triple aim outcomes, a few unfavorable effects on costs, and many inconclusive results."

2. The authors say that a study of 20 PCMH sites would have to reduce costs by 45% or more for the full panel, 20% or more for chronically ill patients only, to establish statistical significance.

But a close read of the paper yields important insights from the authors that explain the context and reasoning behind those conclusions.

It’s not that PCMHs don’t work—it’s that many of the reports/pilots didn't meet statistical standards for "proof"

The most important thing to remember about the study is that it wasn't an evaluation of the PCMH itself--it was a study of the evidence base for PCMH effectiveness--which is not the same. 

For example, of all the pieces ever written about PCMH, the Mathematica researchers only evaluated those that met some starting criteria: 

  • Fitting the researchers’ definition of a medical home (i.e., meeting several of the PCMH principles--important because many "medical home" studies really only target a piece of the medical home, not the whole model)
  • Providing quantitative results on cost, quality, etc.

While the researchers started with 498 studies or articles that arguably showed PCMH value, only 14 of those (based on 12 interventions or pilots) even qualified. In other words, 97% of the articles and so forth that we routinely see extolling the value of the medical home didn't count.  They might still be right about the positive impact of a given PCMH pilot—but the pieces didn't describe controlled studies rigorous enough to support conclusions about the statistical significance of any findings.

Since the vast majority of studies were excluded, one might question whether their criteria might have been too restrictive——but our team has read many of these studies, and  realistically, we would agree that most of the material we see about PCMHs is, essentially, anecdotal. Again, many studies with quantitative results just look at one aspect of the medical home, such as putting in care managers, or e-prescribing, rather than the entire medical home model with multiple principles in place.  Also, many pieces of supporting evidence for the PCMH are before-and-after snapshots of outcomes for a single practice.  From a statistician’s perspective, there is a too-big-for-comfort chance that the results might have been random, or caused by some external factor, not (just) the PCMH transformation.

For statistical purposes, practices are too few in number

Even where studies did include multiple physician practices, comparing outcomes at some sites versus other sites, using physician practice sites as the unit of observation makes a lot of PCMH studies unreliable—again, from a statistical perspective. As the Mathematica researchers point out, in most studies, only a handful of practices are available to make up the intervention or the control groups.

A sample that small is the reason why the researchers have set such a high bar: 45% or 20% demonstrated improvements (depending on the patient population), minimum. The researchers make clear that in a sample of 20 practices, a 10% improvement certainly might be real; but a statistician would not be confident enough in the result to say "medical homes work."

What about the tremendous impacts documented by WellPoint, CCNC, and Geisinger?

One question that is a little unclear about the Mathematica study is whether it also looked at the studies that use claims data to measure the PCMH effect, making the unit of observation PMPM trends, not practice outcomes. (The Mathematica study says it focused only on "practice-level interventions";one article about the study said that researchers did consider at least one claims-based study and had issues with the use of the control group.)

Either way, measuring impact on a PMPM basis is what groups like WellPoint, CCNC, Geisinger are doing. They are not looking at outcomes at one set of physician practices versus a comparison set of practices. From a statistical perspective, that probably puts them into a much better position to measure how effective the model has been over time.

For example, this week’s report on Geisinger’s savings were actually measuring the effect of exposure to the Proven Health Navigator (PHN) program over time—so they were able to use each member’s per-month charges as one unit of observation, adding up to quite a lot of units of observation in the intervention and control group over 4 years. In that way, they were able to say that the PHN program reduced total spending for those members by 7.5% compared to spending on non-PHN members and they were able to demonstrate 95% confidence for their results.

What about shared savings?

A related question that one Medical Home Project participant asked: If these PCMHs have to demonstrate 20% savings or more, how can it be valid for payers to distribute shared savings to Accountable Care Organizations (ACOs) on the basis of a 2% difference in cost savings for a system?

Our experts in the Medicare Payment Innovation Project and Medicare Breakeven Project say it comes back to the unit of observation—because CMS is not measuring site-level performance per se.

  • The unit of observation in a shared savings program is patients—organizations have to demonstrate a per-capita reduction in costs across a very large sample of patients, as compared to a national average cost growth rate (which is a tremendously large control group).
  • At least one track of CMS’s shared savings program does reflect the fact that some organizations have a smaller total sample of patients, and a smaller sample may be less reliable. In the "upside-risk only" track, ACOs with fewer lives have to demonstrate a larger percentage decrease in cost growth rate than a bigger organization does.

Stepping Back: Are PCMHs effective, or not?

The industry may be moving toward the widespread adoption of the PCMH on the basis of some less-than-rigorous evidence, but that doesn't mean the direction isn't right.  Again, the Mathematica study doesn't find fault with the concept of the PCMH.  In fact, looking at the 12 pilots that did pass muster in their sample, they note that "evaluations of six of these interventions provided rigorous evidence on 1 or more outcomes" within the Triple Aim, and they clearly call the early outcomes "promising."  All of their findings and recommendations for next steps are focused on strengthening the evidence base--not overcoming potential shortcomings of the PCMH model.

As for the Advisory Board, we have mostly covered the case for PCMH in a best practice way, not a statistical way. We have never measured PCMH outcomes, in part because comparable outcomes are so difficult to capture.   But the qualitative evidence for PCMH effectiveness is very strong.  A high-functioning PCMH just makes sense as a delivery-system approach to meeting the goals that providers and payers alike are working toward. 

Within the realm of numbers--though not outcomes-- we did just find some promising statistics through the Medical Home Project's operational benchmarking initiative. We found that self-identified PCMHs are statistically significantly more likely to be providing population management services such as patient self-management support and pre-visit chart review. They are also more likely to have key IT tools like EMR and disease registries in place. Those operational differences alone should translate into quality improvement and utilization-linked cost savings down the road.  (For more information on our benchmarks, please make sure you have signed up to receive blog notices—we will be releasing the full results shortly.)

The real challenge in PCMH evaluation: All PCMHs are not alike

The greatest difficulty facing even the best-structured, largest-scale evaluations of the medical home is that every PCMH is different, and some versions of the model are much more effective than others at inflecting downstream utilization in ways that reduce total cost. 

It's actually quite difficult to assess whether a practice is a "real", functional PCMH from the outside—we work with many PCMHs that are not accredited, and some of them achieve major results in a short time frame, while others may be PCMHs "in name only".  Even among practices that are accredited, even at the same level, there will be sites that go far beyond the accreditation standards in achieving patient outcomes, while others may still be at the "check the box" stage. 

This is why much of the Medical Home Project's coverage of medical homes is best-practice and not truly quantitative—our experience has been that the most useful and practical approach to studying and implementing the PCMH model is to look qualitatively at  concrete, replicable practices common to medical homes.

More information


Amanda Berra