Draft AI Act: EU needs to live up to its own ambitions in terms of governance and enforcement

Going forward with the proposed AI Act, the European Parliament and the Member States should re-think its risk-based approach, focus on affected communities, and beef up transparency requirements and enforcement mechanisms. We very much welcome the EU’s efforts to develop a framework for the governance of AI-based systems based on European values and the protection of fundamental rights, but there’s a long way to go to achieve these objectives.

Jump to the summary of our submission here or download the full text as PDF:

Summary

Today, we submitted our response to the European Commission’s consultation on the Artificial Intelligence (AI) Act. The Commission’s proposal has the potential to profoundly shape regulation of AI-based and automated decision-making (ADM) systems in the next decades, not only in Europe but across the globe—through its direct legal extraterritorial effects as well as through its political implications. We appreciate the agenda-setting character of the proposal, which will likely stimulate the urgently needed debate on the governance of ADM systems.

However, we fear that in its current version, the draft AI Act would not reliably and comprehensively accomplish its stated objectives. We call upon the Council and the Parliament to take appropriate measures to correct its shortcomings, clarify its ambiguities, and enhance its consistency, ultimately turning the Act into an effective means for using ADM systems to the benefit of people—and not to their detriment.

We therefore recommend the following:

Better Define the Far-Reaching and Broad but Porous Scope of the Act

Instead of regulating an AI-based system based on its belonging to a specific type of technology, the impact such system has on individuals and society should be the decisive factor in the Act’s regulation. The concept of automated / algorithmic decision-making systems would more precisely capture this aspect.

Mitigate the Self-Defeating Potential of the Risk-based Approach

Make sure that the Act’s harmonization efforts do not result in ADM systems being scrutinized less thoroughly. Member States must have the right to subject such systems to additional requirements in order to mitigate their potentially harmful impact. It should be mandatory for every system deployed by public or private actors to conduct an impact assessment, allowing to determine their respective risk levels on a case-by-case basis.

Make Sure Risk-Assessment is More than a Fig Leaf

Ensure that the standardization procedures are conducted in a transparent, inclusive, and democratic way, including experts and civil society, in order to ensure they indeed do contribute to the protection of the values the Act is based on. Make sure that there is a comprehensive conformity assessment that is not rendered void, cannot be circumvented, takes place within an unambiguous governance structure, and adequately includes third-party oversight.

Make Use of the Potential of the EU Database as a Step Toward Greater Transparency

The database should not only include high-risk systems but—with regard to the public sector—be complemented by a list of all ADM systems in use by public authorities, regardless of their assigned risk level. Complement the information required in Annex VIII with the purpose of the system, an explanation of the model (logic involved), and details on the actors involved in developing and deploying the system, as well as the results of any algorithmic impact assessment.

Clarify Enforcement Mechanisms and Equip Bodies with the Necessary Means

Clarify the roles of the entities involved in enforcement. These entities must be sufficiently independent, adequately resourced, and have the relevant expertise—in both technology and fundamental rights—to fulfill the tasks assigned to them.

Focus Accountability Frameworks on Those Affected

Introduce legally binding data access frameworks, focusing explicitly on supporting and enabling public interest research, which are in full respect of data protection and privacy law. Ensure easily accessible, affordable, and effective legal remedies for affected individuals and groups to contest automated decisions.

Effectively Draw a Red Line on Biometric Mass Surveillance

Comprehensively ban all uses of biometric recognition systems in public space that can lead to mass surveillance and that are, therefore, inherently in conflict with fundamental rights.

Guarantee Participation and Enhance Capacity to Protect Workers’ Autonomy

Guarantee workers the right to obtain information about the purpose of systems covered under this Act, and how the system’s purpose is intended to be achieved. The Act must include process requirements for participation and co-determination options to guarantee that workers do not become mere objects of a “governance by algorithm”.

Find AlgorithmWatch’s full position statement here.