From Theory to Practice: Where do Algorithmic Accountability and Explainability Frameworks Take Us in the Real World
at ACM FAT* 2020, 29 January, 15:00-16:30 and 17:00-18:30, Room: MR7
Moderators and Presenters:
Fanny Hidvegi (Access Now), Anna Bacciarelli (Amnesty International), Daniel Leufer (Mozilla fellow hosted by Access Now) [1st part]
Katarzyna Szymielewicz (Panoptykon Foundation), Agata Foryciarz (Stanford University), Soizic Penicaud (Etalab), Matthias Spielkamp (AlgorithmWatch) [2nd part]
This joint session will take academic concepts and their formulation in policy initiatives around algorithmic accountability and explainability and test them against real cases. Our long term objective includes the assessment of the functionality of these frameworks and models according to experts on the one hand, and in comparison to the expectations of people outside the academic and policy bubble such as end users, regulators and civil society organisations.
While a lot of work has been invested in developing methods, metrics and frameworks in the fields of accountability and explainability [see a list of References below], we face a gap between these sophisticated concepts and much less advanced policies and practices. The second tier of the problem with this gap is that neither these academic concepts nor the known policy frameworks have entered the broader public debate about what it means on a daily basis to explain that a decision was made on the basis of some form of automated information processing. Coming from civil society and public administration, we see our role in filling this gap and confronting concepts and frameworks developed in academic networks such as FAT with the needs that we identify in the outside world.
While most conversations about “Artificial Intelligence” are painfully stuck in between voluntary ethics guidelines, sandboxing for innovation, and demands for the application of human rights frameworks, governments and international organisations are rushing to write AI applications into laws, to boost AI uptake, and to come up with some form of regulation. In the meantime, systems that implement statistical, machine learning, optimization, or autonomous computing techniques are being used by private and public sector entities and they have an impact on individuals and our societies.
Therefore in this hands-on session we will (1) test selected frameworks on algorithmic accountability and explainability against a concrete case study (that likely constitutes a human rights violation) and (2) test our own assumptions regarding how important aspects of an AI system (such as input data, type of an algorithm used, design decisions and technical parameters, expected outcomes) can and should be explained to various stakeholders (affected communities, watchdog organisations, clients).
We invite participants with various backgrounds: technologists, human rights advocates, public servants, industry representatives and designers.
- AI Now Institute "Algorithmic Accountability Policy Toolkit" AI Now Institute (2018).
- Diakopoulos, Nicholas, et al. "Principles for accountable algorithms and a social impact statement for algorithms" FAT/ML (2017).
- Doshi-Velez, Finale, and Been Kim. "Towards a rigorous science of interpretable machine learning" arXiv preprint arXiv:1702.08608 (2017).
- Doshi-Velez, Finale, et al. "Accountability of AI under the law: The role of explanation." arXiv preprint arXiv:1711.01134 (2017).
- Gilpin, Leilani H., et al. "Explaining Explanations to Society" arXiv preprint arXiv:1901.06560 (2019).
- Huk Park, Dong, et al. "Multimodal explanations: Justifying decisions and pointing to the evidence" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.
- Molnar, Christoph. "Interpretable machine learning" (2018).
- Rudin, Cynthia. "Algorithms for interpretable machine learning" Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 2014.
- Wachter, Sandra, Brent Mittelstadt, and Chris Russell. "Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GPDR." Harv. JL & Tech. 31 (2017): 841.
background on the case:
- Ireland National Children's Hospital Chooses Hikvision End-to-End With Facial Recognition
- The FOI request
- COUNCIL OF EUROPE - Unboxing Artificial Intelligence: 10 steps to protect Human Rights - Selection of the “Do’s and Don’t’s”
- HLEG key guidance
- Toronto Declaration excerpts
We will pick one very specific and well-defined AI-related case study to test it against a number of key frameworks that allegedly offer some guidance and criteria for assessing whether that AI-enabled system is “trustworthy”, “responsible”, or ethical according to these frameworks definitions for accountability and explainability.
In doing this, we have two objectives: first, to see if the different policy tools lead to different or similar assessment of the case with regard to its human rights impact, and secondly, to identify potential gaps and omissions between these frameworks and the practical implications and applications of AI. We hope to test our existing toolbox of risk and impact assessment methods to ultimately see where current frameworks are falling short.
Some examples of the policy tools that we will consider using as a framework all include principles around fairness, accountability and transparency:
- An ethics guidelines: On Monday 8 April, the European Commission’s High Level Expert Group on Artificial Intelligence (HLEG) published its “Ethics Guidelines for Trustworthy AI”. The concept of trustworthy AI is introduced and defined by the Guidelines as a voluntary framework to achieve legal, ethical, and robust AI. Alternatively, we would pick an ethics guidelines developed by a private sector actor.
- AI Now’s algorithmic impact assessment model
- A human rights based, normative framework: the Council of Europe’s Unboxing artificial intelligence: 10 steps to protect human rights or the Toronto Declaration
- Guide for Algorithmic Auditing (January 2020), produced by Eticas R&C on behalf of the AEPD - if it’s publicly available and shared in advance
The format allows carefully selected experts from different fields to use their knowledge to lead group discussion. The format builds on engagement and emphasises diversity and participation in order to develop more reliable estimates of the suitability of AI frameworks to our picked use case. The goal is explicitly to generate a high level of diversity of viewpoints, stakeholders and geographical perspectives, and to challenge the concept that AI is ‘too difficult’ to create concrete regulatory policy and accompanying practical guidelines around. Our expert moderators will facilitate the group discussions in an inclusive manner that promotes diversity of opinion and helps to include newcomers to the debate. The group discussion will also enable other participants to contribute and shape the workshop discussion. Finally, we plan a brief debate in the second half of this part of the session to present an opportunity for any participant to become an active member of the workshop in presenting the outcomes of their group discussions and allows all participants’ votes to contribute to the group consensus-based confidence estimates. The selected case and frameworks will be circulated in advance of the session for the invited experts and promoted online for other potential participants.
In the second part of the workshop we will discuss, using concrete examples, what information about the design of AI systems is meaningful to various groups of stakeholders, how it should be communicated to meet the needs of these groups and what are the limitations in asking for “complete transparency” of AI systems.
We will kick start this session by presenting a simple flowchart (see below) explaining key human decisions that shape an AI system and determine its impact:
After this short presentation, we will divide participants in 4-5 groups according to their interests. In smaller groups we will dive deeper into the needs of various stakeholders who, for different reasons, are interested in understanding how an AI system functions:
- Technologists and data scientists, who design such systems;
- Representatives of watchdog organisations, who want to keep these systems accountable
- Representatives of affected individuals, who want to protect their rights
- Clients from private sector and public administration, who need to make sure that an AI system does what it was supposed to do.
During the session we hope to receive input and feedback from various stakeholders, including public administration, industry, researchers and activists regarding: (i) their needs, when it comes to transparency and accountability of AI systems; (ii) their concerns and (iii) best practices they are willing to share. In the final part of the session we will collect key take-aways from small groups and discuss them in the plenary.
- Agata Foryciarz, Daniel Leufer and Katarzyna Szymielewicz, Black-Boxed Politics: Opacity is a Choice in AI Systems
- ICO and Alan Turing Institute, Draft Guidance: Explaining AI decisions
- Timnit Gebru and Jamie Morgenstern and Briana Vecchione and Jennifer Wortman Vaughan and Hanna Wallach and Hal Daumé III and Kate Crawford, Data Sheets for Datasets
- Mitchell, Margaret and Wu, Simone and Zaldivar, Andrew and Barnes, Parker and Vasserman, Lucy and Hutchinson, Ben and Spitzer, Elena and Raji, Inioluwa Deborah and Gebru, Timnit, Model Cards for Model Reporting