AI governance is underfunded in healthcare

In a new report from Black Book Research, researchers surveyed 182 hospital leaders between Oct. 15 and Nov. 8, 2025, about their current AI governance strategy.

In the report, many respondents highlighted how critical governance is for the success of AI initiatives. For example, 70% said they had experienced at least one AI pilot failure due to weak endpoints, workflow misalignment, or data gaps. Eighty percent of respondents also said it was difficult to verify vendors' AI claims without formal governance.

However, the median share of hospitals' 2026 IT quality or safety budgets allocated to AI governance and safety was only 4.2%. Large health systems allocated a median of 6.8% to AI governance while small hospitals allocated only 2.3%.

Even with higher spending at large health systems, many organizations continue to lag on their AI governance policies. Currently, only 29% of respondents said they have implemented and enforced AI policies around model inventory, lineage, and sign offs. Around half (48%) are still drafting their AI policies.

This lack of investment in AI governance has also impacted hospitals' ability to effectively audit AI initiatives. Only 22% of respondents said they could produce a complete AI audit trail for regulators or payers within 30 days. Confidence varied depending on the size of the organization, with large medical centers reporting 34% confidence and small hospitals reporting 15% confidence. 

Commentary

According to Black Book, health systems with an AI governance council are two times more likely to achieve ROI within 12 months. Programs with dashboards and clear ownership also reach early ROI in roughly 7.5 months compared to 13.5 months of those without.

To help hospitals and health systems improve their AI governance, Black Book recommended four actions:

1.       Fund a full AI governance stack with model registry, lineage, monitoring, and override logs.

2.       Contract for audit rights in all vendor contracts to ensure explainability artifacts are delivered.

3.       Clarify ownership of AI governance across IT, quality/safety and compliance, with quarterly reports to the board.

4.       Conduct a 30-day AI audit drill in the first quarter of the year and address any identify gaps.

"The healthcare industry is entering the accountability era of AI," Brown said. "Boards and executives can no longer rely on enthusiasm and vendor marketing alone. Governance defined, operational, and transparent is now the essential infrastructure for safe, equitable, and financially justified AI."