The meaning of algorithmic unfairness or algorithmic bias is often unclear. The term bias has a specific meaning in statistics but “algorithmic bias” is used, nowadays, with much wider meaning, than that.[1] Typically, machine learning algorithms include a ground truth or true label that is used to run statistical tests. Algorithmic bias also includes the bias that affects the so-called “true label”. Suppose you train an algorithm to guess the grade a student would have gotten, in an exam the student will not be able to take. The algorithm will attempt to make this guess on the basis of the patterns it recognizes in the grades that teachers have been giving to students in the past. If these grades are already biased, then even a predictor that is “unbiased” according to a certain statistical standard (using actual grades as the reference) may still count as biased.

To avoid confusion, in what follows, the expression “unfairness” instead of “bias” will be used. So, the theme of this short article is algorithmic unfairness and why it is (or perhaps, should be) such a difficult subject for journalists to cover and write about.

This article introduces a distinction between three types of unfairness. It locates different sites of unfairness and determines, on this basis, whether journalists have a socially valuable role to play in discussing and exposing the unfairness in question, and what such role could be. This is not intended as a full scientific treatment of the question but it offers itself as a primer on the issue.

Unfairness, type 1: Proxies and observation bias

First of all, an algorithm may be unfair because the algorithm uses proxies that unfairly advantage or harm some groups.

Deciding on the basis of proxies is something humans often do. The police may stop individuals with football teams jerseys more often than individuals wearing other types of clothing, in order to check for illegal alcohol detention during a football match where they fear violence against opposing football fans.[2] But clearly, wearing a football jersey is not the cause of buying alcohol and getting violent after a football match. The use of the proxy is harmful for those people who wear a football jersey without the intention of getting drunk, or even to go watch the match. Wearing a football jersey does not cause the risk of getting drunk or being violent at football matches to be higher (rather, the correlation is explained by a common cause, being a football fan, that causes a higher risk of both getting drunk and violent during the match and of buying a football jersey). Wearing a jersey is not like drinking alcohol, which causes an increase in the risk of a car accident, or of violent behavior. Rather, wearing the jersey is causally related to the risk of violence but not itself the cause.  

One often finds considerable societal support for proxy-based decisions, even when they single out specific groups as bearers of disproportionate disadvantages – in the case of the example above, those who wear football jerseys. It is possible that readers believe there is nothing wrong in using this – admittedly coarse – proxy, as the benefits for the population may be high and the cost for the individuals stopped is low.

Yet, it seems that most people’s moral intuition changes when the group disproportionately affected by the proxy is a socially salient group that is victim of injustice and discrimination. This raises the question about the ethical permissibility of algorithms that (1) are fed with data with the intentional goal of finding out accurate (and hopefully robust) proxies for all sort of predictions and (2) cause disproportionate harm to groups that have been, historically, victims of injustice.

One case that has, historically, stirred moral concern and political controversy is redlining in the USA. The area of a city in which a person lives is a good statistical predictor of some forms of financial risk, e.g., the ability to repay a debt. But this is not because living in an area causes some people to be bad at repaying debts. Rather, people who are unequal in their financial risk are more likely to live in similar areas, which however also includes people whose financial risk is different from theirs. As in the case of the football jersey, the proxy is far from perfect. For example, a person with a very secure financial situation may prefer to live near people with the same ethnic, religious or cultural background who may be, on average, in a more precarious existential and financial situation. The area may be used to predict that someone is a bad risk even when it does not cause that person to be worse off financially. This use of proxies is especially problematic in so far as it enhanced racial inequality in society. African-American and Hispanic populations disproportionately lived in the city areas designated to be bad risks. So, for all these people it was much harder to obtain a loan, which further worsened the conditions of the neighborhood. This contributed to further reducing the economic opportunities of people who were already, in their lives, victims of racial discrimination.

Machine learning algorithms designed to achieve accuracy above everything else can learn to use imperfect proxies that are especially damaging to specific groups. Journalists have good reasons to find out about such cases and demand that proxies that are damaging to individuals of disadvantaged groups should not be used. This is especially fruitful in so far as proxies, by definition, are not intrinsically related to the goal the decision-maker needs to achieve. Often, the use of one proxy rather than another is a matter of convenience: the data one can collect determine the range of possible proxies that may be used. So, society needs to put pressure on algorithm designers to use proxies that do not have a negative impact on groups that are already singled out by broader social injustice. The fact that an algorithm (using proxies) is more accurate than previously available prediction methods is not, in and by itself, a conclusive reason to regard that algorithm as morally acceptable.

Unfairness, type 2: Predicting through causes that are affected by group membership

The second type of situation is one in which the feature that is used to make predictions is the cause of risk. In our previous example it is alcohol, not the jersey. But the cause is influenced by group membership, at least to some degree. For example, suppose that rich people enjoy better schools and other training experiences than poor people. Take, for argument’s sake, a very well-intentioned equal opportunity employer, who does not even look at the color of the skin, the nationality, or the name of the school of the prospective employee. She tests all employees on the job, observing their actual skills and problem-solving abilities in operation for six months, then decides whom to hire with a permanent job. A machine learning algorithm also learns from her observation and, over time, learns to make the same judgments about skills the human resource specialist makes during probation, let us suppose, by using the same causally relevant attributes that the HR specialist observes (as opposed to proxies of those attributes).

In this situation, there may still be a form of indirect discrimination, for example, the algorithm may be less likely to select individuals from a minority background. Here one can distinguish different reasons for this effect. First of all, it is possible that the human resource specialist whose judgement the algorithm tries to mimic has a bias against the people from a minority background.

But there is a more interesting type of case, one in which unfairness is produced by the unequal distribution of risk without any form of bias in observation. Suppose that (a) poor people have enjoyed worse educational and training opportunities and (b) people with a minority background are disproportionately represented among poor people. Then, selecting individuals based on how they perform on the job with an impartial test leads to disparate impact independently of whether it is the human or the machine who runs that test. That is to say, individuals are differentially impacted by the test (or by the algorithm that learns to predict its outcome) depending on the group to which they belong. (The term “disparate impact” from US law is perhaps better than the corresponding legal term in Europe, “indirect discrimination”, to describe the case in which the test is, by hypothesis, not biased. For discrimination suggests some kind of injustice in the procedure, while here the problem is with injustice and discrimination in the input that then leads to a discriminating impact although the procedure can be seen as free of discrimination.)

By hypothesis, this is a case in which the selection reflects the social disadvantage of one group relative to another. By assumption, this social disadvantage is reflected, but not produced by the procedure. The procedure legitimately considers the ability that individuals have at that point in their development as workers, and unjust social conditions have affected the probability of each individual to acquire the abilities that employers value. Even with a perfectly fair test, an inequality may still emerge between groups, which may reflect different background opportunities.[3] This will be an inequality between individuals whose merits on the job differ. It is both deserved and undeserved. It is deserved, in so far as people who have different merits on the job may justifiably be treated differently when a hiring decision is made. It is undeserved on account of the fact that many individuals did not deserve the favorable conditions that made it possible for them to acquire the traits employers value in the first place.[4] It is a moral dilemma in which the meritocratic principle pushes in one direction and the equality of opportunity principle pulls in the other.[5]

In a case like this it is not entirely reasonable to blame the algorithm (or its designers) for the unfairness in the decisions algorithms deliver. It seems that what is to blame is the societal arrangement that makes it harder for employees to acquire certain skills and behavioral attitudes that employers value, depending on their social background. The algorithm’s decisions simply reflect those inherent in society.

Should journalists highlight algorithmically produced inequalities in this type of case? Arguably, yes, because it will contribute to stimulate an open public debate about social causes of unfairness. The existence of algorithmic selection makes these inequalities measurable which leads to asking more precise and direct questions about the way our society works and the background injustices it contains. The hard evidence of statistics that algorithms deliver may push societies who want to think of themselves as fair,[6] to ask uncomfortable questions about social justice in society as a whole.

Arguably, journalists may do this without blaming the algorithm or its designers directly. It is plausible that the journalist may take a skeptical stance: just pointing to the inequality, and asking the experts why this emerges, no matter how uncomfortable the replies are (which may vary a lot, and may also reflect the political orientation of the person interviewed).

Unfairness, type 3: Natural inequality

According to some US statistics, Black people are far less likely to develop melanoma than non-Hispanic White people (at a rate of 1 per 100,000 compared to 30 per 100,000) due to the protection that melanin, the body’s natural skin pigment, provides from damaging ultraviolet rays.[7]

This fortunate fact has, predictably, some counterintuitive implication about the prospects of equality for individuals who receive a melanoma diagnosis through a machine learning (or any type of statistical) decision-system. It is predictable that, even lacking any observation bias and with equal technological capacity to retrieve clinically salient information from digital images, Black people with melanoma will be more likely to be misdiagnosed than White people.[8]

To see why, let us consider an extreme (and unrealistic) case. Although the case is not realistic, it can help to grasp this concept at the intuitive level, without mathematical proofs or introducing complex statistical concepts.

Suppose that doctors regard a skin mole to be worthy of surgery only if the risk of it being a malignant mole is equal to or higher than a 0.4 threshold. Now suppose that (hypothetically) the population of black skin moles is made up of moles whose risk of being malignant is below 0.3. Given the risk distribution stated above, every mole diagnosed in a Black person is seen to have a 0.3 risk or lower of being malignant. In contrast, every mole diagnosed in a white person is seen to have a 0.4 risk or higher of being malignant. Given that the risk threshold is determined by doctors as justifying surgery in terms of a positive benefit-risk ratio, every mole in every individual who is black does not satisfy the criterion and every mole in every individual who is white satisfies it. Coherently with the risk criterion in use, not a single mole in a black patient is surgically removed and examined. If one then looks at the population of black individuals with malignant moles, one will notice that the chance of those patients receiving treatment was 0. That is to say: not a single malignant mole was surgically removed. If one looks at the proportion of malignant moles in the white population, one notices that the chance of these malignant moles to be removed was 1. That is to say, every single malignant mole was surgically removed. Is this unjust? Whatever your reply, consider that this is simply the result of making a clinical decision based on risk and of using the same risk threshold for two populations, in which the distribution of risks is very different and far apart.[9]

What should journalists do regarding the third type of case? This is a category of cases in which, arguably, it is not very useful to interpret a statistical inequality (e.g., the rate of false positives and false negatives differs between Blacks and Whites) as a sign of unfairness by the algorithm. The argument here is not that, since a single risk threshold is used, the outcome is clearly fair. But arguably, it is much harder to build an argument that this would be unfair and any such argument is going to meet significant resistance and a good degree of opposition on philosophical, legal, and other normative grounds.[10]

Plausibly, many of these cases of natural inequality in the distribution of risk are encountered in the health sector. It is important that the specificity of those cases is recognized by journalist lest the societal debate about the unfairness of algorithms gets discredited.

Unfairness, type 4: A mix of reasons

A clear complication is that the three sources of unfairness I have distinguished can all materialize in the same case. That is to say, unequal algorithmic predictions may result from a combination of observation bias, non-causal proxies, causal proxies reflecting background social injustice, and some natural inequality. Even worse than that, journalists and the experts they interview may not be in the position to discern which of the causes is in question. Furthermore, “black box” machine learning algorithms can deliver efficient and reasonably accurate “prediction machines” even in domains in which no-one, among the domain experts, can explain why it makes sense to predict some given Y through some observation X.

The overlap of different causes for inequality can explain why the journalistic debate on crime prediction was unconclusive.[11] Consider the event “justified arrest by the police”. This event has two parts: (1) a crime, for which the individual is responsible; (2) someone (from law enforcement) to witness or persecute the crime. Suppose that the unequal risk of individuals relative to the event “justified arrest” is due to the combination of a disproportionate risk for crime for (1) and a disproportionate tendency towards arrest (given a crime of the same severity) for (2). While excluding the hypothesis that one group is more prone to crime for natural causes, as this would be clearly racist, it is possible that the disproportionate risk for crime is due to adverse social circumstances (among which, racism) to which members of the Black population are (disproportionally) exposed, in comparison to Whites.  Because of (2), relying on predictive algorithms to maximize the success of policing may be considered clearly unfair: it is a problem of observation bias in the police, and the algorithm reproduces that bias. But even if this were not the case, the disparity in crime propensity deserves to be critically evaluated in its social causes, even when the causes are not flaws in algorithmic design. This seems especially plausible when the effects of the algorithms (more Black people in prison) contribute to causing those adverse circumstances that increase the crime propensity of one group in the first place.[12]

Here, journalists ought to be guided by their intuition about the interplay of different levels of injustice. As a practical guide, it is useful to distinguish at least four main questions that it may be useful for journalists to consider:

1. Is the algorithm making the prediction using a proxy or something that domain experts recognize as a bona fide cause of the effect Y that one is trying to predict?
2. If the prediction uses a proxy, could a proxy be chosen that has less of an impact on the socially salient groups affected by the decisions, especially those that are historically victims of discrimination?
3. If the prediction uses a cause X of the effect as a means of predicting the effect, is the distribution of X across different sub-groups of the population affected by racism, sexism or other unjust social process?
4. If the algorithm uses both causes and proxies, and both causes affected and not affected by unjust social processes, what would happen to the accuracy of the algorithm by using only the causes, or only the causes less affected by unjust social processes, or by using proxies that are less prone to have a different impact on different socially salient groups (including artificial proxies, that can be generated with the help of algorithmic techniques)[13]?

Given the complexity of cases that are relevant enough to merit journalistic investigation, these questions quite frequently can only be sufficiently answered by teams of journalists with different skills, and will often require consultation of additional experts.