AutoCheck – Mapping risks of discrimination in automated decision-making systems
NEW REPORT: Our AutoCheck Guidebook on discrimination by automated-decision making systems helps identify cases and get the right support. Read now!
Automated decision-making (ADM) processes are emerging in more and more areas of life: They assess our creditworthiness, set individualized prices on online retail platforms, or suggest which movies to watch. However, their use does not necessarily lead to less-biased decisions. ADM systems can perpetuate or even reinforce inequality and discrimination. For example, a study by AlgorithmWatch showed that job ads published on Facebook were displayed to different audiences based on gross stereotypes: A job ad for a truck driver was 10 times more likely to be displayed to men than women, and a job ad for an educator was 20 times more likely to be displayed to women than men. If ADM systems make or suggest discriminating decisions, it potentially affects many people—because all decisions follow the same system-specific pattern. These risks are systemic and the overall societal impact can be large.
“I am sure: Risks of discrimination through automated decision-making systems will encompass all areas of life. It will be one of the most relevant issues in anti-discrimination work.“ (German anti-discrimination expert)
Between February 2021 and July 2022, we developed concrete and comprehensive instructions and tools and also designed training courses in our project AutoCheck – a Guide about Automated Decision-Making Systems for Equality Bodies. Our work was based on case study research and interviews with anti-discrimination experts. The results provide answers to fundamental questions, such as: What are automated decision-making systems? How does discrimination occur? How can I recognize this form of discrimination and what can be done about it? In our publications, you can find case studies and concrete assistance in the form of tips, checklists, sources, and contact information for support.
Our AutoCheck publications primarily address employees at anti-discrimination offices in Germany. By building up competencies in this increasingly important field, we hope to enable them to better recognize and assess risks to support those affected by discrimination. But we would also like to give those affected by discrimination the opportunity to inform themselves about the topic.
You can download our Guidebook "Automated Decision-Making Systems and Discrimination" here.
You can find the complete workshop documents here. The documents are designed in a way that the workshop can be held independently by anyone with some prior knowledge. In case you are interested in having us conduct a workshop for you or your organization, please feel free to contact Jessica Wulf.
Funded by
