Who Taught the Machine to Discriminate?
Investigating Hidden Power in AI Systems Through Data, Algorithms and Reverse Engineering
Time to Complete: 30 minutes
PDF 5-Minute Warm-Up Activity can be downloaded above.
Who This Is For: This lesson is for anyone who works with, evaluates or is affected by automated systems that make decisions about people. That includes policy analysts and civil society advocates working on AI regulation and digital rights who need conceptual tools to interrogate vendor claims about fairness; UX researchers, product managers and data scientists in tech, healthcare, finance and criminal justice who suspect bias in deployed systems but lack a structured method to investigate it; educators, journalists and public interest researchers who communicate about AI and need a rigorous framework that goes beyond anecdotal case studies; and HR professionals, compliance officers and diversity leads in organizations adopting AI hiring or performance tools who must assess whether those systems disadvantage protected groups. The shared problem across all these roles is the same: AI systems are routinely described as objective and neutral, yet the consequences of their errors fall disproportionately on people who have the least power to challenge them. This lesson provides a structured investigation method that does not require access to source code or proprietary datasets.
Real-World Applications
Content moderation platforms and hiring algorithm providers now face regulatory scrutiny, including the EU AI Act and US EEOC guidance, that requires demonstrable bias audits before deployment. The hermeneutic reverse engineering method covered in this lesson maps directly to the audit process those teams must conduct: observe system outputs across demographic groups, identify patterns that suggest differential treatment, hypothesize what data or design choices produce those patterns and document findings in terms that regulators and affected communities can act on. Academic bias research and industry compliance practice are no longer separate tracks, and this lesson bridges them by giving practitioners a research-grade investigative framework that does not assume a computer science background.
Lesson Goal
You will develop critical skills to examine how AI systems encode and reproduce social bias by applying hermeneutic reverse engineering. By the end of this lesson you will be able to identify bias patterns in AI outputs, trace those patterns to data collection and algorithmic design decisions and propose concrete alternatives grounded in inclusion and accountability. The lesson draws on critical data studies, critical algorithm studies and feminist science and technology studies to build a multilevel framework for investigation.
The Problem and Its Relevance
AI systems are not neutral technologies. They are built by specific people, trained on specific data and deployed in contexts that reflect existing social hierarchies, yet they are routinely presented as objective tools whose outputs represent truth rather than design. This gap between the rhetoric of neutrality and the reality of embedded bias is not a minor oversight. It is a structural condition that requires deliberate investigation. The consequences are measurable and direct. Facial recognition systems trained predominantly on light-skinned faces fail to detect dark-skinned faces because the societal definition of a standard face was encoded into the training data from the start. Search algorithms associate criminality with images of Black teenagers while returning wholesome results for comparable queries about white teenagers, encoding patterns that mirror historical discrimination rather than neutral retrieval logic. Treating algorithmic bias as a bug to be patched mistakes the symptom for the disease. The data and code of an AI system do not produce bias by accident; they crystallize the power relations of the institutions and people that designed them, which means the real question is not how to fix the output but who decides what counts as correct in the first place. The communities most harmed by discriminatory AI outputs are also the least represented in the datasets those systems learn from, creating a compounding inequality that technical audits alone cannot resolve. Accountability requires participation, and participation requires power, and the design of current AI development pipelines systemically excludes both.
Why Does This Matter?
Understanding how to investigate AI bias through reverse engineering matters because:
1. Bias is structural, not accidental: Discrimination in AI stems from systematic coding of dominant social norms, not random errors. Recognizing this shifts the focus from fixing outputs to examining design decisions and power structures.
2. Data carries political history: Datasets are cultural narratives assembled by specific people with specific worldviews. When data is disconnected from its collection context, embedded subjective choices are misread as objective facts.
3. Algorithms construct realities: Machine learning systems do not discover truth. They calculate possibilities based on design decisions that favor certain outcomes while excluding others, and each calculation is a political act.
4. Invisible decisions have visible consequences: Every algorithmic choice about data processing and output selection constitutes a value judgment that affects real people, often reinforcing inequalities that already exist.
5. Harm is unevenly distributed: Women, people of color, disabled individuals and queer communities face the greatest impact from AI injustices while having the least access to design and development processes.
6. Documentation gaps enable abuse: Lack of transparency about whose data is collected, why it is collected and for what purpose enables default discrimination and the overexposure of minoritized groups to surveillance.
7. Commercial incentives shape every design choice: The primary purpose of most deployed AI systems is to serve corporate interests rather than user welfare, and this drives decisions from data collection through to output selection.
Three Critical Questions to Ask Yourself
• Can I explain how examining patterns in AI outputs reveals political assumptions embedded in data and algorithms, even when source code remains proprietary?
• Do I understand the difference between individual instances of bias and systemic patterns that indicate structural discrimination built into a system by design?
• Am I able to identify which social groups are included or excluded in a given AI system's design and articulate what power dynamics those exclusions reflect?
Activity Steps
Review the framework of hermeneutic reverse engineering as developed by Balsamo (2011): observe and describe, analyze, interpret, articulate, rearticulate, prototype, assess, iterate, produce, reflect and critique. This lesson applies each step to the investigation of an AI system's data and algorithmic politics. Treat the system you select as a boundary object that exists across multiple disciplines and requires examination through the lenses of critical data studies, critical algorithm studies and feminist science and technology studies.
Working individually or in groups, your task is to:
(i) Select an AI system that makes automated decisions affecting people's lives. Options include hiring algorithms, predictive policing tools, content moderation systems, automated loan approval platforms, healthcare diagnostic tools or facial recognition software.
Guidance: Choose systems where differential treatment across demographic groups has been documented or is plausible given the context of deployment.
(ii) Investigate the data politics of your chosen system by questioning:
◦ Whose data likely trained this system and whose data is absent?
◦ What historical power relations and collection contexts does this data carry?
◦ How does disconnection from original contexts enable biased interpretations of that data?
◦ Who created the categories and classifications that organize the data, and why?
(iii) Examine the algorithmic politics by analyzing available outputs and documented patterns:
◦ What realities does this algorithm create or reinforce among different user groups?
◦ Which groups benefit from these decisions and which groups face harm?
◦ What decisions made during design enabled discriminatory outcomes?
◦ How does the system's commercial purpose shape what it produces?
(iv) Develop a reverse engineering analysis that includes:
◦ Observable patterns in system outputs that suggest bias or differential treatment
◦ Hypotheses about data characteristics or design choices that could produce these patterns
◦ Identification of whose perspectives and experiences the system excludes
◦ Analysis of how the system reflects or challenges existing power structures
(v) Speculate alternative designs by imagining:
◦ What would this system look like if marginalized communities had designed it?
◦ How could different data collection or algorithmic choices produce more equitable outcomes?
◦ Which counter-narratives remain hidden in the current design?
◦ What structural changes would address root causes rather than surface symptoms?
(vi) Propose concrete interventions drawing from established research practices:
◦ Actionable auditing strategies that publicly name biased performance results
◦ Documentation requirements that record data collection context and purpose
◦ Participatory design processes that center the experiences of affected communities
◦ Fairness metrics that address individual and group-level discrimination without requiring collection of sensitive demographic data
Guidance: Balance idealism with pragmatism. Propose interventions that acknowledge systemic constraints while pushing toward more accountable AI futures.
Individual Reflection
After completing the activity, consider:
• How this investigation changed your understanding of whether AI can be neutral or objective
• What you discovered about the relationship between technical design choices and social power
• Whether you will engage with automated systems differently now that you can identify assumptions embedded in them
• How the method you practiced applies to other AI systems you encounter in your professional field
• What responsibilities developers, regulators and users each share for addressing structural discrimination in automated systems
The Bottom Line
Effective bias detection requires moving beyond the assumption that discrimination is a technical error and recognizing it as a political outcome produced by specific choices about data, design and deployment. Hermeneutic reverse engineering succeeds not by accessing proprietary code but by reading observable outputs as social texts that reveal whose worldviews shaped the system and whose realities it ignores or harms. Yet identifying bias is not the same as challenging the conditions that produce it. The auditing tools that researchers and regulators now advocate are valuable, but they can also function as a buffer that absorbs criticism without redistributing the power to decide what AI systems should do and for whom. The demand for explainable AI has generated a new vocabulary of fairness that companies adopt without changing their incentive structures. Transparency reports and bias dashboards are not accountability; accountability requires consequences, and consequences require power that affected communities currently do not have. When you can articulate how a specific AI system perpetuates structural discrimination, identify which communities bear disproportionate harm, trace how technical choices reflect political decisions and envision more equitable alternatives, you have developed the critical literacy needed to challenge algorithmic authority rather than accept it. This understanding applies whether you build AI systems, regulate their deployment, advocate for affected communities or navigate daily environments shaped by automated decisions that claim objectivity while exercising power.
#AIBias #AlgorithmicAccountability #CriticalAIStudies #DataPolitics #AILiteracy