Spam filters are efficient and uncontroversial. Until you look at them.

An experiment reveals that Microsoft Outlook marks messages as spam on the basis of a single word, such as “Nigeria”. Spam filters are largely unaudited and could discriminate unfairly.

Story

22 October 2020

#publicsphere

Nicolas Kayser-Bril
Head of Journalism (on parental leave)

In an experiment, AlgorithmWatch sent a few hundred emails to 10 email inboxes at Gmail, Yahoo, Outlook, GMX and LaPoste (the last two are used by millions of Germans and French, respectively). All accounts were created specifically for the experiment.

The results, which are available online, show that Microsoft Outlook considers the following as spam:

Spam detectors at other providers did not display the same behavior. Outlook was the only provider where we could identify the words that triggered the spam filter.

Microsoft declined to comment. It is unlikely that an Outlook engineer made an explicit rule to mark any message that contains “Nigeria” as spam. Instead, a machine learning algorithm probably identified “Nigeria” as a strong discriminator between spam and non-spam messages. Microsoft does not make the training data set of its spam filter available to researchers.

SpamAssassin’s creed

SpamAssassin is a spam filter developed by the Apache Software Foundation. It is widely used by organizations that maintain their own email servers. Unlike most commercial offerings, SpamAssassin’s code is open-source and can be reviewed.

While SpamAssassin’s rules change daily, its default configuration files single out words like “Ivory Coast”, “Nigeria” or “Nigerian government” as spammy. The phrase “Oprah!”, an African-American entertainer, is listed as potentially spammy, though the rule is currently inactive.

Rules are changed based on daily checks on training data submitted by users. No effort seems to be made to ensure that user-submitted data does not discriminate unfairly.

User-submitted data is not available, but some of the training data sets are. SpamAssassin published a public corpus of spam and non-spam mail (which the anti-spam community calls ham) which, while over 15 years old, is still widely used. In the spam folder, 59 emails out of 1,397 are from Nigerians. In the ham folder, none are.

The SpamAssassin Project Management Committee did not answer our questions but stated that problems with specific rules were managed by “the community”.

White privilege

SpamAssassin’s leadership is aware of the racism and white privilege embedded in software. In July, it announced that its next release would use “welcomelist” and “blocklist” to replace the racially-charged terms that were used until then.

However, while SpamAssassin says that “[they have] a particular self-interest in attracting contributors from a diversity of cultures”, its Project Management Committee seems to be composed exclusively of white men (some members use pseudonyms and could not be verified with certainty). And at least one of its members routinely signs the emails he posts on the SpamAssassin’s mailing list with anti-feminist quotes from a far-right columnist.

Did you like this story?

Every two weeks, our newsletter Automated Society delves into the unreported ways automated systems affect society and the world around you. Subscribe now to receive the next issue in your inbox!

Get the briefing on how automated systems impact real people, in Europe and beyond, every two weeks, for free.

For more detailed information, please refer to our privacy policy.