1. Pre- vs. post-intervention: This analysis compares patients’ results before the intervention to their results after the intervention. While this analysis is simple, it is not sufficient. It fails to control for a variety of factors, including natural regression to the mean.
2. Expected vs. actual: Another common method is to compare expected spend for a patient population without intervention to the observed spend with the intervention using a hypothetical trend rate. Although this method is slightly more accurate, the savings it measures are only theoretical.

While randomized controlled trials are the scientific gold standard, they’re often impractical — or even unethical — in many real-world scenarios.

Intervention vs. “control” group: The best way to measure the effectiveness of interventions

Intervention vs. control analysis, also known as the “concurrent control” method, involves measuring healthcare spend for an intervention group that receives a targeted intervention. These results are compared to the healthcare spend of a control group who received no intervention over the same period — but that control data comes from a large dataset that already exists. The control group should be adjusted for variables such as age, severity, and presence of comorbidities, all of which can drastically impact the care costs for a population. This is often done through methods such as propensity score matching, which is a statistical technique that ensures the distribution of characteristics is similar in both groups by assigning propensity scores and matching them.

An intervention vs. control analysis is the most accurate way to measure return on investment (ROI) for these interventions. Comparing the population who received the intervention to an adjusted control population minimizes the impact of uncontrollable variables. However, this type of analysis is extremely difficult since it requires data from a large number of patients to accurately adjust for confounding covariates of the control group.

When evaluating the impact of interventions, health plan actuaries expect not only sound methodology but also credible data sources — and many providers and vendors do not have access to this. This is why having an evaluation partner who can structure high-quality data through the health plan lens is crucial to building credibility.

What to look for in an effective evaluation partner

Evaluation partners can help organizations prove the value of their interventions. Here’s what makes a good one.

1. Large, high-quality dataset (e.g., claims data). Proving that a clinical intervention or care management program is effective requires a significant amount of data and analysis across multiple years. Large, long-term datasets allow analysts to (1) adjust for confounding factors such as age and severity of the control population and (2) measure the full impact, which may take many months after the clinical intervention. For example, several studies measuring the impact of GLP-1s and other therapeutics have fallen short because datasets aren’t equipped to measure medium- to long-term effects. Any dataset used to support the analysis needs to span multiple years to be able to capture patients’ longitudinal claims patterns, as well as account for health plan member turnover and churn.
2. Deep actuarial expertise to meet plan requirements. An effective evaluation partner should have the capabilities to structure analyses (e.g., robust methodology, relevant data points) that meet health plan expectations. Health plan actuaries are often risk-averse in how they approach new business with partners and vendors. If an organization wants to engage in value-based care (VBC) contracts, it needs robust statistical evidence to support claims about the value of the interventions. Having the foundation of the analysis created through a health plan's lens speeds up the contracting process.
3. Strategic view. Developing an effective communication strategy is necessary to prove the overall value of an intervention to a health plan. For a VBC payment model, articulating how the intervention decreases the total cost of care (TCOC) is essential but difficult. A good evaluation partner can develop a communication strategy that addresses the financial environment and depicts the overall value of the intervention.

Oshi Health is a virtual multidisciplinary gastrointestinal (GI) clinic. It offers a dedicated team of licensed GI experts who diagnose and help patients navigate the full range of GI conditions such as ulcerative colitis, Crohn’s disease, and undiagnosed GI symptoms. Oshi’s integrated team of specialists work collaboratively in a high-touch model to diagnose and treat patients to get them better, faster. These specialists include GI-specialized advanced practice providers, registered dietitians, gut-brain specialists, and care coordinators — all overseen by gastroenterologists.

Oshi Health provides patients with unlimited virtual visits and support between visits based on patients’ needs. With an average of 8 to 10 provider visits and 85+ meaningful touchpoints per patient, Oshi’s care teams can iterate on treatments such as medications, GI behavioral therapies, and dietary interventions to help patients reach symptom control. Oshi also coordinates with in-person GI groups and primary care groups bidirectionally when needed.

Oshi Health leaders recognized gastrointestinal care as a high-cost category with little existing precedent for a compelling risk model. Data on GI costs wasn’t widely studied. And GI care is particularly complex as GI costs widely vary depending on the patient. Some patients only visit a PCP once when they start to experience GI symptoms while other patients escalate to a series of specialist visits, diagnostic tests and imaging, and potentially ED visits — which drive up costs.

In addition, separating GI disease modalities is critical, as patients with Crohn’s disease may have significantly different cost profiles than patients with irritable bowel syndrome. Failing to appropriately account for these variables may result in the analysis understating Oshi Health’s impact, putting them in a disadvantageous position in negotiations. Alternatively, not accounting for the variables could overstate Oshi Health’s impact, undermining the credibility of the study.

Oshi Health partnered with Optum Advisory, Provider Actuarial Services to run a series of intervention vs. control analyses. These analyses demonstrated the clinical and financial effectiveness of the GI interventions across all of Oshi Health’s member cohorts.

• 96% of members achieved symptom control
• 89% of members reported reduced symptom severity
• 48% of members assessed with a new GI diagnosis
• 62% reduction in GI-related emergency room visits [p=0.0003]
• 42% reduction in GI-related imaging [p=0.0000]
• 54% reduction in GI-related surgery [p=0.0468]
• $6,081 savings in total cost of care per patient [p=0.0019] resulting in 5.9x ROI

Optum Advisory, Provider Actuarial Services was an effective evaluation partner to Oshi Health by providing the necessary datasets and analysis to engage in a successful risk-based payment model with commercial health plans. Oshi Health effectively demonstrated that its interventions and treatments reduce utilization and improve outcomes for patients. Proving positive financial and patient outcomes with a credible evaluation partner significantly increased Oshi Health’s value proposition to health plans. Additionally, Oshi Health developed targeted and data-informed value-based contracts, with a mixture of downside risk and fee-for-service (FFS). With contracts built around outcome measures informed by the intervention vs. control analyses, Oshi Health has continued to outperform their contracts under this risk-based payment model.

Conclusion

Understanding and proving the financial impact of TCOC interventions is crucial for securing payer support and ensuring sustainable healthcare solutions. By using robust data, actuarial evidence, and a trusted partner (when needed), leaders can demonstrate the true value of their interventions, ultimately leading to better contract terms and improved patient outcomes.