Building fairer data systems: Lessons from addressing racial bias in healthcare

In an era where data-driven decision-making increasingly shapes our world, ensuring fairness and equity in these systems has never been more critical. The training of AI models and other data systems has, at times, been responsible for perpetuating historical biases in the outputs of the latest data systems.

A report by the World Economic Forum’s Global Future Council on Data Equity proposes a new framework for defining and implementing data equity as a response to the risk of perpetuating old biases in new technology.

A startling example of such bias came with the analysis of an algorithm used in the US healthcare system. The research revealed the algorithm had been systematically disadvantaging Black patients in need of complex care.

The root cause of this lay in the data used to train the algorithm, which reflected the fact that Black patients historically received less expensive treatments due to income inequality and other barriers to healthcare access.

The study concluded that correcting this bias could increase the percentage of Black patients receiving additional care from 17.7% to 46.5%. This case underscores the critical need for rigorous algorithm auditing and cross-sector collaboration to eliminate such biases in decision-making processes.

There is currently no globally agreed standard or definition of data equity. In order to move forward on solving the problem of bias, the Forum’s Global Future Council on Data Equity has drawn up a comprehensive definition:

"Data equity can be defined as the shared responsibility for fair data practices that respect and promote human rights, opportunity and dignity. Data equity is a fundamental responsibility that requires strategic, participative, inclusive and proactive collective and coordinated action to create a world where data-based systems promote fair, just and beneficial outcomes for all individuals, groups and communities. It recognizes that data practices – including collection, curation, processing, retention, analysis, stewardship and responsible application of resulting insights – significantly impact human rights and the resulting access to social, economic, natural and cultural resources and opportunities.”

This definition highlights a crucial point: data equity isn't just about the numbers. It's about how those numbers impact real people's lives. It encompasses the entire journey of data, from how it's collected and processed to how it's used and who benefits from its insights.

Implementing data equity requires a structured approach. The data equity framework, developed by the Global Future Council, offers a tool for reflection, research and action. It's built on three main pillars: data, people and purpose.

The three core pillars of the framework break down as follows:

The nature of the data: This pillar focuses on how sensitive the information is and who can access it. For instance, health records are highly sensitive and require strict protection.

The purpose of data use: This pillar aims to consider factors like trustworthiness, value, originality and application. Is the data being used ethically? Does it provide genuine value to society?

The people involved: This addresses the relationships between data collectors, processors, and subjects and the responsibilities that come with data handling. It also considers the expertise of those handling the data and, crucially, who's held accountable for its use.

If we return to the example of problematic healthcare data above, we can see how these principles might be applied. The Global Future Council’s report has the following initial recommendations to remove bias.

Building fairer data systems is not a one-time fix – it's an ongoing journey that demands monitoring, collaboration and a continued, unwavering commitment to equity at every step. The lessons we're learning from addressing racial bias in healthcare algorithms can and should inform how we approach data equity across other sectors.

Contributing authors: Simon Torkington, Senior Writer, Forum Agenda and Stephanie Teeuwen, Specialist, Data Policy and AI, Centre for the Fourth Industrial Revolution, World Economic Forum.