Algorithmic Injustice: When AI Systems Make Biased Decisions

Artificial intelligence systems are becoming deeply embedded in our lives, influencing everything from which social media posts we see to whether we get approved for a loan. While AI promises greater efficiency and objectivity, these systems can inherit, amplify, and perpetuate human biases. The lack of diversity among AI developers, flawed training data, and focus on metrics over social impact all contribute to unfair and discriminatory algorithmic decisions. This article explores what algorithmic injustice looks like, why it happens, and what can be done to ensure AI promotes justice and inclusion.

The Rise of Algorithmic Decision-Making

Algorithms are everywhere in our digital lives, though often invisible. An algorithm is simply a set of programmed instructions or rules that a computer follows to solve a problem or complete a task. AI algorithms utilize machine learning, allowing systems to improve through experience over time. As AI is integrated into more critical systems, these algorithms are increasingly making impactful decisions with real-world consequences.

AI now determines which candidates get interviewed or hired for jobs by analyzing video interviews and resumes. Algorithms decide who sees content in their social media feeds and who gets targeted with online ads. Risk assessment tools predict who is likely to commit a future crime or default on a loan. AI is being used in healthcare to diagnose illnesses, in education to score exams, in criminal justice to set bail amounts, and more.

While AI is often praised for its speed, scalability, and seeming objectivity, algorithms can unfairly discriminate against certain groups. Just as human decision-makers are prone to prejudice, the data and rules which train AI systems often embed society’s biases. This can lead to marginalized groups being adversely impacted by algorithmic injustice.

What is Algorithmic Injustice?

Algorithmic injustice or unfairness occurs when an AI system produces discriminatory, unethical, or biased decisions impacting certain groups more than others. This stems from flaws in how algorithms are developed, trained, and deployed in ways that disproportionately harm minorities, people of color, lower-income groups, and other marginalized populations. Even with good intentions, algorithms can deny opportunities, resources, and rights to those who already experience structural inequity.

Some examples of algorithmic bias include:

• Facial recognition software that misidentifies people of color more often than white people
• Hiring algorithms that rate female candidates lower than male ones with identical qualifications
• Healthcare algorithms that provide less accurate diagnoses and treatment recommendations for Black patients
• Risk assessment tools that label minority offenders as higher risk, leading to harsher sentences
• Advertising algorithms that show STEM career ads more frequently to men than equally qualified women
• Ride-hailing apps that charge higher fares in low-income neighborhoods

While some amount of bias is inevitable in human-created systems, excessive harms that reinforce discrimination and widen inequality are ethically unacceptable, legally dubious, and bad for business. Though AI creators may have good intentions, algorithmic injustice persists due to:

• Lack of diversity – The majority of AI developers are white men, leading to blind spots
• Data biases – Training data reflects societal biases and lack representation of minorities
• Metrics focus – Focus on predictive accuracy over social impact

Without proactive changes, algorithmic injustice will worsen as AI spreads to more high-stakes domains like healthcare, employment, finance, and the justice system.

Key Causes of Algorithmic Injustice

AI algorithms designed without diverse perspectives, trained on flawed data, and laser focused on metrics over social good can amplify social biases, discriminate unfairly, and cause harm – even absent ill intent. Here are the key factors behind algorithmic injustice.

Lack of Diversity in AI Development

The lack of diversity among AI researchers, developers, and company leadership is a major contributor to algorithmic bias. The overwhelmingly white, male homogeneity leads to groupthink, blindness to the lived experiences of marginalized groups, and failures to consider fairness, accountability, transparency, and social impacts.

Key data points:

• Only 14% of AI researchers at Facebook and 10% at Google are women. Less than 3% of AI researchers are Black or Latinx.
• 85% of data scientists are men and 72% are white according to the Burtch Works 2022 salary study.
• 15% of executive positions at major tech companies are held by women and just 3% by underrepresented minorities according to the Kapor Center.

When the engineers building these systems all have similar backgrounds, key perspectives are excluded and social biases creep in unnoticed and unchallenged. Having greater diversity would surface different viewpoints, reveal overlooked issues, and prompt deeper thinking around potential harms early in the design process.

Biases Encoded in Training Data

AI systems are trained on vast amounts of data meant to represent the world around us. However, data generated by society does not accurately reflect reality but contains embedded biases against minority groups and women. Relying on this data cements unjust structural inequalities into automated decisions.

Key examples of biased training data:

• Facial recognition datasets with an overrepresentation of lighter skin tones lead to misidentification of darker skinned faces.
• Hiring algorithms trained on data of past candidates reflect historical biases, with white men disproportionately hired.
• Medical datasets with fewer examples from minorities contribute to treatment algorithms being less accurate for marginalized populations.

Addressing this requires proactively seeking balanced and unbiased datasets, augmenting with data from underrepresented groups, and continuously evaluating models for potential harms.

Metrics Focused on Accuracy Over Fairness

In machine learning, models are optimized to maximize metrics like accuracy, speed, and profitability. While important, focusing solely on these metrics fails to account for social responsibility, ethics, and fairness – which are much harder to quantify and codify.

An accurate but unjust algorithm is not really accurate for society as a whole. For example:

• A hiring algorithm can accurately predict job success by favoring majority groups, but this makes it biased against minorities with equal potential.
• A medical diagnosis algorithm may be highly accurate overall, but less so for underrepresented minorities not well reflected in the training data.

Building fair and ethical AI requires going beyond metrics to deeply consider social impacts on the most vulnerable and traditionally marginalized groups who may not be fully represented in the data.

Impacted Groups and Real-World Consequences

While algorithmic bias can impact anyone, marginalized and minority groups bear the greatest consequences. Being denied opportunities by automated systems reinforces long-standing social inequities. Here are examples of real-world impacts on affected groups:

Racial Minorities and People of Color

• Wrongful arrests due to racial bias in predictive policing algorithms
• Facial recognition leading to false matches for people of color
• Being deemed higher risk by criminal risk assessment tools, leading to harsher sentences
• Seeing fewer job ads due to racially biased ad targeting algorithms

Women and Gender Minorities

• Being offered lower salaries or hiring ratings by algorithms inheriting gender bias
• Seeing gender-stereotyped job ads and content by ad algorithms
• Transgender users deadnamed by facial recognition apps
• Receiving less accurate medical diagnoses and treatment plans from algorithms trained on male-dominated data

Low-Income Groups

• Being shown higher-priced products and services online
• Being deemed higher credit risks due to associations between income and risk in data
• Fare hikes and removal of shared bikes from low-income neighborhoods by rideshare algorithms
• Less access to loans, housing, healthcare, and education opportunities predicted by biased algorithms

Other Marginalized Groups

• Disabled users encountering inaccessible design overlooked by non-disabled developers
• Neural networks amplifying offensive content about marginalized groups when trained on unfiltered web data
• LGBTQ+ users deadnamed by apps using legal name databases with outdated identity information
• Neurodiverse users like those with ADHD underserved by learning and wellness algorithms optimized for typical brains

While not inherently malicious, these examples show how algorithmic injustice manifests in ways that worsen social inequity and limit advancement for marginalized communities. Well-intentioned companies can inadvertently cause harm with AI.

Social Consequences of Algorithmic Injustice

The biases and harms from AI systems have far-reaching societal consequences beyond just the individuals directly impacted. A few key issues include:

• Perpetuating historical discrimination into automated decisions
• Contributing to and exacerbating existing social biases
• Widening gaps in wealth, health, education, accessibility, and equality between groups
• Reducing diversity by creating additional barriers for underrepresented groups
• Losing out on innovations and collective benefits by denying opportunities to marginalized voices
• Normalizing unfairness and making discrimination seem objective and technological
• Diminishing public trust in technology companies and AI systems deployed unethically

Algorithmic injustice also raises critical questions about accountability, ethics, governance, and human rights – like who is responsible when AIs discriminate, and how we balance innovation with protecting the most vulnerable and equitable access to technology’s benefits.

How to Ensure Greater Algorithmic Justice

Making AI ethical, fair and socially responsible requires:

1. Inclusive Development
2. Unbiased Training Data
3. Holistic Evaluation Metrics
4. Proactive Audits
5. Strong Governance

Let’s look at how companies, researchers, and policymakers can enact change in each of these areas.

1. Inclusive Development

Who builds technology matters. Having greater diversity among AI teams leads to:

• More voices at the design table spotting potential issues
• Lived experiences that improve understanding of marginalized needs
• Reduced homogeneous groupthink and blindspots

Key steps to take:

• Improve representation of women, minorities, and people with diverse backgrounds among AI teams at all levels
• Include sociologists, ethicists, humanists, and civil rights experts in the development process
• Create more educational opportunities in AI for marginalized groups underrepresented in STEM
• Seek community input and participate in public-private collaborations around ethically aligning AI with societal values

1. Unbiased Training Data

Data is never neutral – it contains social biases and gaps in representation. Companies must:

• Proactively identify underrepresentation and overrepresentation of social groups within training datasets
• Collect more data representative of marginalized populations, with their consent
• Synthetically generate complimentary datasets for improved balance
• Continuously review and address data issues contributing to algorithmic unfairness

1. Holistic Evaluation Metrics

Looking beyond performance metrics to prevent myopia, companies should:

• Formally evaluate AI systems for unwanted biases using rigorous auditing processes
• Collect feedback directly from impacted groups and listen to issues faced by marginalized communities
• Define metrics not just for accuracy but fairness, accessibility, accountability, transparency, and impact on human rights
• Openly research, report, and publish metrics related to AI ethics, safety, and algorithmic justice

1. Proactive Audits

Being proactive takes more effort but prevents reactive PR crises around algorithmic bias. Steps for companies:

• Perform extensive bias and discrimination testing prior to product launch using varied demographics
• Implement data tracking to monitor for real-world performance gaps indicating potential bias against users
• Correct issues through prompt model re-training, patch fixes, or temporarily disabling AI features if needed
• Institutionalize periodic third-party auditing and external ethical reviews

1. Strong Governance

Laws, policies, and self-regulation establishing collective accountability and oversight for ethical AI are key. Progress requires:

• Governments expanding anti-discrimination laws and protections to algorithmic systems
• Technology companies committing to internal processes, dedicated roles, and executive accountability around AI ethics
• Academic and non-profit research shedding light on algorithmic harms and best practices
• Stronger public awareness and pressure on tech firms to address algorithmic injustice and be transparent

The Way Forward

AI has immense potential for improving lives but also risks exacerbating social inequities if deployed unethically. Ending algorithmic injustice requires those building this transformative technology to embrace values of diversity, equity, and inclusion; consult communities affected by their systems; and take collective responsibility for social impacts.

Companies serious about avoiding algorithmic bias must go beyond platitudes and make meaningful internal changes to data practices, development processes, evaluation metrics, governance policies, and culture. Individuals in tech concerned about fairness in AI can advocate for these actions from within and publicly pressure change through resignations, leaks, and organizing if faced with organizational inertia.

Policymakers must also modernize regulations to extend anti-discrimination protections to automated decisions, increase corporate accountability, and ensure public transparency. Impacted communities should have a voice in the use of AI systems that affect their lives. And diverse innovators who have known exclusion must help lead the development of technology that uplifts all people.

Together, through a shared commitment to algorithmic justice, we can harness the powerful potential of AI to build a fairer future guided by the highest human values of justice, empowerment, and the dignity of all.