Why does this matter? 

Poor design choices can both exacerbate existing inequities for patients and create new ones. AI has the potential to perpetuate human and societal biases that would further marginalize minority communities in healthcare. Understanding the various ways in which inequity shows up in AI can help your organization actively avoid mistakes that further inequity.

Ways bias manifests in AI design

The data and algorithms that shape AI models are influenced by humans with conscious and unconscious biases. Those designing AI solutions can overlook or miss biases in the data, make assumptions about the people using the AI model, or fail to anticipate unintended consequences of an application. Moreover, AI models can unknowingly be trained on ingrained historical biases and stereotypes already present in healthcare processes and data.

A study examining the relationship between race and receiving pain medication in an emergency department (ED) found that risk of receiving no pain medication was 66% greater for Black patients.¹ A similar study measuring the relationship between pain management and race for children with appendicitis in the ED found that Black patients with moderate and severe pain were less likely to be prescribed pain medication.² An AI model built to predict the need for pain medication could therefore reinforce inequities in pain treatment for Black patients if the developers used such data.

Personal biases, historical biases, stereotypes, and poor data quality all contribute to algorithmic bias. Algorithmic bias is more likely to be perpetuated when there is a lack of diversity among the coders, programmers, and stakeholders involved in designing and evaluating AI models. The absence of diverse demographics and perspectives increases the chances of overlooked biases and makes the AI model more applicable to individuals who share a similar background with the model's designers.³

What can you do about it?

Organizations should take these actions to avoid bias in model design. Each step is meant to build off the previous one. 
 

• Ensure a demographically diverse team of programmers and coders. Individuals from different backgrounds (race, gender, sexual orientation, ability, etc.) are more likely to consider the implications of an algorithm on people who share similar backgrounds.3 If you’re partnering with vendors, ask about their diversity, equity, and inclusion efforts and the diversity of their programmers. Not doing so will produce costly long-term consequences such as lawsuits, smaller consumer segments, and loss of credibility due to potentially discriminatory AI models.
• Involve a diverse group of stakeholders in evaluating AI model design. Health systems and vendors should collaborate with health equity leaders, clinicians, and patients to gain insights on how AI can best serve your diverse patient population. Health equity leaders possess extensive knowledge of your patient’s needs and the most effective ways to distribute patient-facing AI solutions equitably. Clinicians can identify potential biases and ensure that the solution aligns with clinical best practices. Patients can also identify biases and pain points that impact their use of the AI solution.
• Design algorithms with diverse and representative training data. Algorithmic bias appears when the training data for a model does not match the patient population it serves. Developers of AI models must integrate a wide range of data that represents patients’ demographic, socioeconomic, and geographic backgrounds. Techniques such as oversampling, undersampling, or data augmentation can be used to balance the representation of different groups in the training data. AI models can reduce biases against underrepresented groups by including a wide range of diverse and representative data.
• Conduct a thorough review of your training data. Racism, stereotypes, and historical biases can inadvertently be incorporated into AI models through various sources such as biased literature, journals, internet sources, and people. Train developers to critically examine literature and data to prevent biased resources from entering the training data. While a large quantity of data is important, it should not come at the risk of introducing bias into AI.
• Conduct regular bias audits, monitoring, and evaluation. Regular bias audits involve evaluating the AI system's outputs for potential biases across different demographic groups. By addressing biases early on, you can make necessary adjustments to improve the fairness and accuracy of the AI model. Continuous monitoring and evaluation of the AI model’s performance is necessary to mitigate biases that may emerge over time. This involves analyzing the system's performance across different demographic groups, monitoring for disparate impacts, and making necessary adjustments to ensure equity.