Stories

How does automated decision-making effect our daily lives? Where are the systems applied and what happens when something goes wrong? Read our journalistic investigations on the current use of ADM systems and their consequences.

27 April 2021

#surveillance

Greek camps for asylum seekers to introduce partly automated surveillance systems

An EU-funded surveillance system for “reception and identification centers” on five Greek islands raises questions about asylum seekers’ privacy and well-being. Despite assurances from European authorities, the Centaur system suggests that mass control, and not shelter, is the priority.

Ralph / Flickr

15 April 2021

How French welfare services are creating ‘robo-debt’

When automated fraud detection algorithms fail, welfare services can wrongly demand the repayment of benefits. Over the last five years, several scandals showed the breadth of the problem. In Australia, 400,000 people were put in ‘robo-debt’, 40,000 in Michigan and 26,000 in the Netherlands. Journalist Lucie Inland explains how the French welfare office automatically put her in debt, and how she fought back.

Read more
Fraktion DIE LINKE |Flickr.com

6 April 2021

Europeans can’t talk about racist AI systems. They lack the words.

In Europe, several automated systems, either planned or operational, actively contribute to entrenching racism. But European civil society literally lacks the words to address the issue.

Read more
Icons8 | Unsplash

29 March 2021

Automated translation is hopelessly sexist, but don’t blame the algorithm or the training data

Automated translation services tend to erase women or reduce them to stereotypes. Simply tweaking the training data or the models is not enough to make translations fair.

Read more
Roel Wijnants |flickr

3 March 2021

Politicians can do well on Instagram. Political posts, less so.

An experiment by AlgorithmWatch, Groene Amsterdammer, Pointer and NOS in the Netherlands shows that Instagram probably prioritizes images of faces and pushes down images that contain text.

Read more
Piero Nigro | Unsplash

25 February 2021

In Berlin, Google Maps and TomTom encourage car drivers to disregard the law

Routing services from TomTom and Google Maps encourage car drivers to use cycle streets, which is illegal. Both companies said they would update their databases, but this is unlikely to completely solve the problem.

Read more

1 February 2021

The Insta-mafia: How crooks mass-report users for profit

Groups of teenagers use weaknesses in the notification systems of Facebook and Instagram to take over accounts and harass users. The upcoming DSA regulation addresses the issue but is likely to fall short.

Read more
Εφημερίδα ΠΡΙΝ | Flickr

28 January 2021

Flush with EU funds, Greek police to introduce live face recognition before the summer

Greek police are due to receive gear that allows for real-time face recognition during police patrols. Despite concerns that the system could seriously affect civil liberties, details about the project are scarce.

Read more

19 January 2021

Medical devices using AI/ML are poorly regulated: study

A review of 338 AI-powered medical devices approved in Europe and in the United States reveals holes in the European review process.

Read more
Pedro Serapio |Pixabay

12 January 2021

China’s social credit system was due by 2020 but is far from ready

Six years after the government announced plans for a national social credit score, Chinese citizens face dozens of systems that are largely incompatible with each other. The central government is planning an overhaul.

Read more
Robert Scoble | Flickr

6 January 2021

In Poland, a law made loan algorithms transparent. Implementation is nonexistent.

Since May 2019, and as a first in the EU, Polish consumers have the right to know in detail why a bank decided to grant or refuse them a loan, even for small amounts. But in practice, banks are still reluctant to provide such information. 

Read more
Wynand van Poortvliet | Unsplash

18 December 2020

New report highlights the risks of AI on fundamental rights

The European watchdog for fundamental rights published a report on Artificial Intelligence. AlgorithmWatch welcomes some of the recommendations, and encourages a bolder approach.

Read more
foodwatch | Flickr

14 December 2020

Despite transparency, the Nutri-Score algorithm faces strong resistance

The Nutri-Score summarizes basic nutritional information on a 5-letter scale. Despite its many qualities, it faces a strong backlash that could hold a lesson for operators of automated systems.

Read more

4 December 2020

Health algorithms discriminate against Black patients, also in Switzerland

Algorithms used to assess kidney function or predict heart failure use race as a central criterion. Continue reading the story on the AlgorithmWatch Switzerland website

Read more
Wynand van Poortvliet | Unsplash

25 November 2020

Dutch city uses algorithm to assess home value, but has no idea how it works

In a seemingly routine case at the Amsterdam court of appeal, a judge ruled that it was acceptable for a municipality to use a black-box algorithm, as long as the results were unsurprising.

Read more
Jean-Noël Dollé |flickr

23 November 2020

French tax authority pushes for automated controls despite mixed results

Since 2014, a team of data-scientists supports local tax offices to help them identify complex fraud. But the motive could be more base: to make tax collectors redundant.

Read more
Ed Westcott|Wikimedia

27 October 2020

Spanish police plan to extend use of its lie-detector while efficacy is unclear

Veripol is a software that assesses the veracity of complaints filed with the Spanish national police. It was introduced in 2018, but it’s unclear if it works as intended.

Read more
the blowup |Unsplash

22 October 2020

Spam filters are efficient and uncontroversial. Until you look at them.

An experiment reveals that Microsoft Outlook marks messages as spam on the basis of a single word, such as “Nigeria”. Spam filters are largely unaudited and could discriminate unfairly.

Read more
Joshua Hoehne | Unsplash

18 October 2020

Automated discrimination: Facebook uses gross stereotypes to optimize ad delivery

An experiment by AlgorithmWatch shows that online platforms optimize ad delivery in discriminatory ways. Advertisers who use them could be breaking the law.

Read more
Erika Fletcher |Unsplash

18 September 2020

In French daycare, algorithms attempt to fight cronyism

In many cities, it is unclear whose children can hope for a place in a public daycare facility. Algorithms could make the allocation of places more transparent, but not all politicians are happy.

Read more
Jon Russell |flickr

17 September 2020

Female historians and male nurses do not exist, Google Translate tells its European users

An experiment shows that Google Translate systematically changes the gender of translations when they do not fit with stereotypes. It is all because of English, Google says

Read more
elimirana |Unsplash

16 September 2020

In Italy, an appetite for face recognition in football stadiums

Right before the pandemic, the government and top sports authorities were planning a massive deployment of face recognition and sound surveillance technologies in all Italian football stadiums. The reason? To help fight racism

Read more
Catherine Poh Huay Tan |flickr

14 September 2020

Suzhou introduced a new social scoring system, but it was too Orwellian, even for China

A city of 10 million in eastern China upgraded its Covid-tracking app to introduce a new “civility” score. It had to backtrack after a public outcry.

Read more
Jilbert Ebrahimi|Unsplash

25 August 2020

For researchers, accessing data is one thing. Assessing its quality another.

Online platforms often provide data that is riddled with errors. Rather than launching quixotic attempts at fixing them, researchers increasingly investigate why platforms bias their data.

Read more
Jehyun Sung |Unsplash

24 August 2020

GPT-3 is a lot of fun, but no game-changer

We usually do not write about newly-released software, especially when there is no way to audit it. But the hype over GPT-3, a natural language generator, was such that several readers asked for a review.

Read more
Kamil Gliwiński |Unsplash

18 August 2020

Pre-crime at the tax office: How Poland automated the fight against VAT fraud.

In their fight against fraud, Polish tax authorities use STIR, an algorithm sifting through the data of millions of entrepreneurs. The government claims success, but dozens of companies have been hit, some say wrongly.

Read more
Thomas Stephan |Unsplash

13 August 2020

Under the Twitter streetlight: How data scarcity distorts research

As part of our #LeftOnRead campaign, several researchers testified to the reluctance of online platforms to provide useful data. Many resort to studying Twitter, which is more accommodating than most.

Read more
Ivan Aleksic |Unsplash

12 August 2020

Algorithmic grading is not an answer to the challenges of the pandemic

Graeme Tiffany is a philosopher of education. He argues that replacing exams with algorithmic grading, as was done in Great Britain, exacerbates inequalities and fails to assess students' abilities.

Read more
Luis García |Wikimedia

11 August 2020

Spain’s largest bus terminal deployed live face recognition four years ago, but few noticed

Madrid South Station’s face recognition system automatically matches every visitor’s face against a database of suspects, and shares information with the Spanish police.

Read more
Darth Liu |Unsplash

6 August 2020

In a quest to optimize welfare management, Denmark built a surveillance behemoth

Udbetaling Danmark was created in 2012 to streamline the payment of welfare benefits. Its fraud control algorithms can access the personal data of millions of citizens, not all of whom receive welfare payments.

Read more
Artiom Vallat |Unsplash

31 July 2020

Broken Horizon: In Greece, research in automation fails to find applications

Greek researchers led hundreds of research projects that involved automation, but very few found their way into real-life uses. Opinions differ on where the innovation pipeline is clogged.

Read more
Boehlich |Wikimedia

22 July 2020

Swiss police automated crime predictions but has little to show for it

A review of 3 automated systems in use by the Swiss police and judiciary reveals serious issues. Real-world effects are impossible to assess due to a lack of transparency.

Read more
Morning Brew |Unsplash (edited)

9 July 2020

Left on Read: How Facebook and others keep researchers in the dark

Internet platforms routinely deny researchers access to data or arbitrarily accede to their requests, hampering social science in the process.

Read more
Ronnie Macdonald|flickr

7 July 2020

Slovenian police acquires automated tools first, legalizes them later

The Slovenian police legalized its use of face recognition 5 years after it started to use it. Despite formal safeguards, no institution can restrain the Interior ministry.

Read more
Halacious |Unsplash

29 June 2020

Portugal: Automated verification of prescriptions helped crack down on medical fraud

Portugal’s national health service introduced a centralized, automated system to verify medical prescriptions in 2016. One year later, it flagged 20 million euros’ worth of fraud.

Read more
Marcus Kauffman|Unsplash

17 June 2020

Can AI mitigate the climate crisis? Not really.

Several institutions claim that AI will contribute to solving the climate crisis, but evidence is scant. On the contrary, AI has a track record of helping emit more greenhouse gases.

Read more
NeONBRAND | Unsplash

15 June 2020

Undress or fail: Instagram’s algorithm strong-arms users into showing skin

An exclusive investigation reveals that Instagram prioritizes photos of scantily-clad men and women, shaping the behavior of content creators and the worldview of 140 millions Europeans in what remains a blind spot of EU regulations.

Read more

3 June 2020

Ten years on, search auto-complete still suggests slander and disinformation

After a decade and a string of legal actions, an AlgorithmWatch experiment shows that search engines still suggest slanderous, false and disparaging statements.

Read more
Acount222 |Wikimedia

26 May 2020

Estonia: A city is automating homes to reduce energy consumption

The city of Tartu installed automated systems in old housing blocks. Using nudges, sensors and automated decision-making, it hopes to reduce energy consumption by two-thirds.

Read more

19 May 2020

Automated moderation tool from Google rates People of Color and gays as “toxic”

A systematic review of Google’s Perspective, a tool for automated content moderation, reveals that some adjectives are considered more toxic than others.

Read more
Alexandr Bormotin |Unsplash

15 May 2020

Finland: How to unionize when your boss is an algorithm and you’re self-employed

A group of Finnish couriers launched the Justice4Couriers campaign in 2018. Although they are technically self-employed, they must obey the whims of their platform’s algorithm. They are fighting back.

Read more

14 May 2020

This man had his credit score changed from C to A+ after a few emails

A 52-year-old man in Hanover, Germany, discovered that he’d been erroneously scored by a credit bureau. His story reveals the gaps in credit score regulation.

Read more
Luis Villasmil |Unsplash

28 April 2020

Unchecked use of computer vision by police carries high risks of discrimination

At least 11 local police forces in Europe use computer vision to automatically analyze images from surveillance cameras. The risks of discrimination run high but authorities ignore them.

Read more
Melanie Wasser|Unsplash

27 April 2020

In Spain, the VioGén algorithm attempts to forecast gender violence

As part of a program to curb feminicides, Spain built VioGén, an algorithm that assesses the risk faced by victims of gender violence. It remains a work in progress.

Read more
Nicepear Jakarta|Unsplash

15 April 2020

Credit scores algorithms keep operating normally even as everything else doesn’t

Consumer organizations are calling for a credit score freeze but credit bureaus see no need to change anything as the Covid-19 pandemic wreaks havoc in the finances of many.

Read more
Benjamin Davies |Unsplash

8 April 2020

Brexit: How EU nationals navigate the automated checks of the “Settled Status” program

The British Home Office deployed a mobile app for EU citizens to obtain permission to stay in the UK. Some are confident about the partly automated process, others live Kafkaesque experiences.

Read more
Marko Bugarski|Unsplash

7 April 2020

Google apologizes after its Vision AI produced racist results

A Google service that automatically labels images produced starkly different results depending on skin tone on a given image. The company fixed the issue, but the problem is likely much broader.

Read more
Zicht op kruising Europaweg en Boerhaavelaan in Haarlem.
Jeroen van Lieshout |flickr

6 April 2020

How Dutch activists got an invasive fraud detection algorithm banned

The Dutch government has been using SyRI, a secret algorithm, to detect possible social welfare fraud. Civil rights activists have taken the matter to court and managed to get public organizations to think about less repressive alternatives.

Read more
Sint-Ursula-Instituut
Sint-Katelijne-Waver |flickr

3 April 2020

In Flanders, an algorithm attempts to make school choice fairer

In Belgium, some schools don’t have enough capacity for all students that want to go there. In the Flemish part of the country, the government introduced an algorithm to assign places in schools, in the hope of giving every student the same chances.

Read more
Charlotta Wasteson |flickr

17 March 2020

Central authorities slow to react as Sweden’s cities embrace automation of welfare management

Trelleborg is Sweden’s front-runner in automating welfare distribution. An analysis of the system's source code brought little transparency – but revealed that the personal data of hundreds was wrongly made public.

Read more
François Genon|Unsplash

19 February 2020

EU Commission publishes white paper on AI regulation 20 days before schedule, forgets regulation

In a set of documents published today, the European Commission unveiled its strategy for “Europe's digital future”. Despite some promising regulatory proposals, the Berlaymont’s program for automated systems fails to seriously appreciate the risks.

Read more
Médiathèque Pierre Fanlac de Périgueux

11 February 2020

Between care and control: 200 years of health data in France

The French “Health Data Hub” will soon offer health data on all French citizens to AI startups that request it. It is the latest step in a project to centralize health information that began 200 years ago and kept oscillating between care and control, but mostly control.

Read more
Praveesh Palakeel|Unsplash

6 February 2020

The algorithm police is coming. Will it have teeth?

Lawmakers in several European countries are passing or considering legislation to keep algorithms in check. How it will be enforced, and by whom, is still unclear.

Read more
Tim Cole |Unsplash

Interview, 6 February 2020

“Lawmakers should provide rule-based descriptions of what it means not to be racist”

Cathy O’Neil is a mathematician and the author of several books on data science, including Weapons of Math Destruction, which looked at the impact of algorithms on society. In an e-mail interview, she explained how her business, ORCAA, audits algorithms.

Read more
ev | Unsplash

11 December 2019

At least 11 police forces use face recognition in the EU, AlgorithmWatch reveals

The majority of the police forces that answered questions by AlgorithmWatch said they use or plan to introduce face recognition. Use cases vary greatly across countries, but almost all have in common their lack of transparency.

Read more
Husky | Wikipedia

Interview, 9 December 2019

Dutch MP Kees Verhoeven wants a registry of “heavy” algorithms – but it shouldn’t be public

On September 10, Dutch MP Kees Verhoeven put forward a motion in the Dutch parliament, together with MP Harry van der Molen, to create a mandatory register for all public-sector algorithms. He explains how this register could be implemented.

Read more
Fikri Rasyid |Unsplash

4 December 2019

Price-fixing algorithms can come, competition authorities are ready, they claim

In a new report, the competition authorities of France and Germany reviewed how algorithms could lead to cartels and price-fixing. They offer solutions, too.

Read more
Andrew Ebrahim |Unsplash

26 November 2019

New Swiss algorithm to desegregate schools, one block at a time

Two researchers from Zurich University created an algorithm that helps desegregate schools by slightly changing the boundaries of each school’s catchment area.

Read more
Matt Hoffman |Unsplash

Op-ed, 25 November 2019

Controversial service that ranked job seekers based on personal emails folds following AlgorithmWatch investigation

A Finnish company that automatically parsed the personal emails of job applicants to assess their corporate “fit” discontinued its service after reports by AlgorithmWatch and others raised questions about its legality.

Read more
Hanna Morris | Unsplash

12 November 2019

“Explainable AI” doesn’t work for online services – now there’s proof

New regulation, such as the GDPR, encourages the adoption of “explainable artificial intelligence.” Two researchers claim to have proof of the impossibility for online services to provide trusted explanations.

Read more
Parker Coffman | Unsplash

11 November 2019

Palantir, the secretive data behemoth linked to the Trump administration, expands into Europe

The data analysis company, known in particular for running the deportation machine of the Trump administration, is expanding aggressively into Europe. Who are its clients?

Read more
Nik Shuliahin | Unsplash

4 November 2019

Facebook enables automated scams, but fails to automate the fight against them

Scammers massively use Facebook’s advertising platform using so-called “cloakers” to evade automated checks. They would be very simple to detect but, despite announcements to the contrary, Facebook seems to tolerate them.

Read more
Linh Do | Flickr

16 October 2019

UN special rapporteur on digital technology and social protection denounces a “human rights free-zone”

The report of the UN special rapporteur on extreme poverty and human rights condemns in strong terms the way governments are privitizing and automating welfare management.

Read more

6 October 2019

Austria’s employment agency rolls out discriminatory algorithm, sees no problem

AMS, Austria's employment agency, is about to roll out a sorting algorithm that gives lower scores to women and to the disabled. It is very likely illegal under current anti-discrimination law.

Read more
Jonathan Borba | Unsplash

3 October 2019

Ethical guidelines issued by engineers’ organization fail to gain traction

The world’s largest professional association of engineers released its ethical guidelines for automated systems last March. A review by AlgorithmWatch shows that Facebook and Google have yet to acknowledge them.

Read more

2 October 2019

Defective computing: How algorithms use speech analysis to profile job candidates

Some companies and scientists present Affective Computing, the algorithmic analysis of personality traits also known as “artificial emotional intelligence”, as an important new development. But the methods that are used are often dubious and present serious risks for discrimination.

Read more
Nikola Johnny Mirkovic  | Unsplash

12 August 2019

Spain: Legal fight over an algorithm’s code

Half a million Spaniards have been denied an electricity subsidy although they might be eligible. Non-profit Civio is going to court to obtain the source code of the software making the decisions.

Read more
Pablo García Saldaña | Unsplash

20 June 2019

“Ethical AI guidelines”: Binding commitment or simply window dressing?

In April this year, we began to compile an overview of guidelines for so-called “ethical AI”. Thanks to contributions of our readers, our inventory has grown to 83 entries and is finding global resonance. It is therefore time for an interim overview: there are some substantial contributions, but many entries are lacking substance. Additionally, there is a dearth of oversight mechanism to ensure that these guidelines are adhered to.

Read more
Daan Stevens | Unsplash

11 June 2019

Bad data and health: Garbage in, carnage out

French investigative outlet Mediacités recently revealed how design failures in the user interface of software used in the country’s second-largest hospital endangered patients’ lives. Such software is used to gather the data that feeds machine learning algorithms, raising questions on the efficiency of artificial intelligence solutions in the health sector and what they mean for patients' health.

Read more
Marie Bellando-Mitjans | Unsplash

16 April 2019

Poland: Government to scrap controversial unemployment scoring system

The Polish government has been forced to scrap a controversial scoring system for the unemployed after criticism by campaigners, judges and a human rights watchdog. The automated system is used to make life-changing decisions about what support individuals get based on their personal data and answers to interviews at job centres. Critics say the system is discriminatory, lacks transparency and infringes data protection rights. As a result the government has decided to end its experiment with profiling the unemployed, and the system will be scrapped by December 2019.

Read more
Alireza Badiee  | Unsplash

28 February 2019

Sweden: Rogue algorithm stops welfare payments for up to 70,000 unemployed

Automated decision-making has become a national talking point in Sweden, after a report by the country’s public broadcaster revealed thousands of unemployed people were wrongly denied benefits by a government computer run amok.

Read more
Alireza Badiee  | Unsplash

25 February 2019

Sweden: Rogue algorithm stops welfare payments for up to 70,000 unemployed

Automated decision-making has become a national talking point in Sweden, after a report by the country’s public broadcaster revealed thousands of unemployed people were wrongly denied benefits by a government computer run amok.

Read more

21 November 2018

Finnish Credit Score Ruling raises Questions about Discrimination and how to avoid it

In a case not well known outside Finland, a tribunal threatened to fine Svea Ekonomi AB for 100,000 euros on the grounds that it declined credit to a customer by using statistical models that are in violation of anti-discrimination laws. The case raises some hard questions about the practice of credit scoring and how it could be done.

Read more

4 July 2018

High-Risk Citizens

A software is used for detecting potential benefit cheats in the Netherlands. The government keeps quiet about how that works. Civil rights activists are taking the matter to court.

Read more