Why we need to audit algorithms and AI from end to end

We therefore want to share our thoughts on what we mean by this “End-to-End Auditing” of algorithmic or AI systems and why we take this approach, using some specific examples we are considering to investigate further. An idea to keep in mind is that a complicated picture can be seen as a jigsaw puzzle: it can be broken down into many parts, which many people can work together to fit into the right places.

The End-End-Chain

Behind every AI output is a long chain of actions, technologies, and decisions. From deciding on the purpose of a tool; planning and scoping the tool; through to data collection; training and fine-tuning the models; creating interfaces for users; creating and using outputs; to building them into a business model; and iterating the tools following their deployment. There will be trade-offs throughout, policies and decisions to be made, and safeguards and limits which could implemented. Different actors will be involved, and various risks and harms can arise. Thus, before we further explain what we mean by end-to-end, we first have to make clear that we do not understand “auditing AI” as a mere technical process or approach. Rather, the diverse nature of the potential harms should always be audited in relation to the societal context they are manifesting themselves in, requiring diverse perspectives and expertise.

To narrow the scope, let’s focus on one particular form of AI – Generative AI (“GenAI”). These systems can create new content (text, images, audio, video) based on user-defined inputs. Let's consider risks and harms arising at either end of the chain. Let’s start by considering an early “upstream” part of the value chain: fine-tuning the models so they can recognize and avoid producing harmful content. This is known as “de-toxifying” the models. Failure to do so properly led to some early and high-profile PR disasters, such as Microsoft’s Tay Chatbot in 2016 which became famous for tweeting racism and conspiracy theories. While it is certainly possible to get newer chatbots, such as ChatGPT, to produce such content, it is considerably harder - due to de-toxification.

One of the most effective ways to de-toxify at scale is to outsource to humans, who look at content and label whether it is “harmful” – which could include content deemed to be offensive, violent, etc. – so that a model is trained to “recognize” features of this content. As reported by news outlets such as Time Magazine, companies including OpenAI and Meta had outsourced this to a company called Sama. Sama was employing people in Kenya to do the labeling. Some had been hired under false pretenses (job adverts claimed to offer call center work) and had moved to Kenya for the work. The content they labeled, for USD 1.30 to 2 per hour, included harms up to and including extreme (sometimes sexual) violence. Here we see upstream decisions prioritizing speed and cost-saving of model training over ensuring the well-being of humans – potentially even violating those workers most basic rights. 

We could point to further upstream issues, in particular the use of energy and water required to train models. There is increasing talk about energy costs and CO2 emissions, including in AI “model cards” designed for transparency. But as made clear by the UN Sustainable Development Goals, and explored in our own SustAIn project, this does not capture all dimensions of sustainability – environmental (e.g., water, resource mining, disposal), social (e.g., workers' rights), and economic (e.g., power concentration and oligopolies in AI markets).

Now let’s consider one downstream risk of GenAI - the use of tools to create sexualized and degrading images of real women (one could think of other examples). This has been seen in examples ranging from high-profile figures, like Taylor Swift or Giorgia Meloni, Members of Parliament in Germany and Switzerland, through to young women at school. Not only is this inherently awful as well as a potential violation of a person’s right to dignity, but in some circumstances, this could also contribute to the ongoing harassment of women out of important roles in public life, mental health problems for young people, and a range of other systemic risks that governments across the world are aiming to address.

Intervening along the chain

Clearly, there is no one solution for the very different harms AI creates, but many of the issues can be addressed using specific measures. As an advocacy organization, we believe that well-created and properly enforced legislation, which genuinely draws on the input of civil society, can create protections and accountability. This is not the only answer, of course. For instance, the African Content Moderators Union and the Data Workers Inquiry support the workers who detoxify AI models, including by raising the visibility of their experiences and bringing lawsuits against the companies involved. Organizations like HateAid and the Apolitical Foundation are supporting women who are at risk of technology-enabled abuse. However, legislation can help, and it is certainly the case that the EU is trying to provide a range of tools to protect against tech-enabled harms. Using the metaphor of a jigsaw: It’s all about finding the suitable piece that fits in the right place, while still not losing sight of the big picture and see how the pieces should all fit together.

Relevant tools may not even be in the most obvious legislation. For the above case studies, one might immediately consider the AI Act. However, as we at AlgorithmWatch have explained before, the AI Act has multiple gaps concerning addressing risks to fundamental rights. Instead, the forthcoming Corporate Sustainability Due Diligence Directive (also known as the Supply Chain Law) may be better placed to address exploitative labor use in the creation of GenAI tools deployed in the EU. The EU’s proposed new legislation addressing violence against women and girls can, and should, account for harms facilitated by GenAI; in the meantime, the Digital Services Act may also be an appropriate tool to study and address the spread of such content. By keeping the end-to-end chain of potential harms in mind, we can therefore expand the range of relevant tools available for different contexts. This can help involve relevant actors with varying expertise – whether in human rights, the organization of labor, or protecting women and girls – to intervene at different points along the value chain. But this involves careful coordination and communication with groups who may be very distant from technology legislation, and already very overstretched.

Power imbalances and lack of accountability

An end-to-end approach can also consider how the problems are interrelated, and how they are all related to the big picture behind: the massive power concentration and lack of accountability behind AI. For instance, in the cases above, de-toxifying the models can play a positive role in reducing abilities to produce degrading content against women (as well as other groups). However, this cannot be used to justify the indiscriminate exploitation of the workers training the models. Balancing the risks at different ends of the supply chain allows us to consider fairer and more just ways to support those doing the de-toxifying work while ensuring that de-toxifying is carried out effectively. To give another example, borrowing from the legal scholar Phillip Hacker, perhaps certain “downstream” uses of AI - for instance, tools shown to be medically useful - can justify certain levels of resource consumption in the upstream production of models while other uses must be held to much stricter standards.

To rigorously consider and enforce the above-outlined balancing acts requires reliable and transparent information. Which brings us to our final point. Looking across end-to-end chains of GenAI can help us identify common problems – most obviously, the imbalances of power between the companies and everyone else, and the lack of transparency and accountability, which will bedevil any approach to solve them and where unified fronts will be most needed, as the European Centre for Non-Profit Law have recently laid out. In our jigsaw puzzle, many of the most important pieces are being hoarded by Big Tech companies (and, to a certain extent, regulators). Other pieces are being fought over by too many groups, like the disproportionate interest paid to “deep fakes” and elections. Some pieces may come from groups with little engagement in the tech policy world, yet who have the experience and expertise to contribute to these discussions. These are all practical challenges on which civil society organizations and others must carefully coordinate.

Building the jigsaw

If this sounds like a complex picture – well, it is. But this is the world that big tech companies have handed us by releasing GenAI in the manner they have. We need to face this problem as it currently exists and not try to reduce it to overly simple standardized frameworks. We cannot scrutinize it at just one level by making models “explainable”, for instance, or by red-teaming in laboratories. The trick is to break it down into smaller ideas, which, when added together, show a bigger overall picture: the political economy behind AI that is dominated by a handful of billion dollar tech companies. So we should break this down into smaller ideas, while at the same time not lose sight of this bigger, overall picture of what is in and behind the technology that is currently being presented to us.

At AlgorithmWatch we combine broader research and advocacy work – be that around the Digital Services Act, the AI Act, surveillance, or other topics – with specific empirical projects, such as our work on chatbots and elections or AI at the borders of the EU, and investigative journalism. Concerning the two examples outlined above – exploitation in de-toxification and degrading content against women – we are exploring how to investigate these in ways that reveal the opportunities for intervening “along the value chain” with a range of tools. We are approaching partners who work in these fields and are considering potential participatory research approaches. And, as is often the issue with NGOs, important ideas don’t pay for themselves and we must find ways to sustainably resource such work. We believe it is important that complex, seemingly abstract issues like “end-to-end auditing” can – and must – be addressed by looking at the real-life issues faced by people experiencing the harshest ends of technology.