CRAFT: Critiquing and Rethinking Accountability, Fairness and Transparency (ACM FAT* 2020)

CRAFT aims to “bring parties together that may not always share a common space or common values, and that are situated in different power structures” and therefore announced a call for submissions. CRAFT welcomes contributions that address key critiques of the field, including blind spots, omissions and alternative possibilities and that open new lines of inquiry, collaboration, and practice.

We very much endorse CRAFT’s holistic approach, as we have argued since AlgorithmWatch’s inception that automated decision-making processes must not be regarded as technological tools but need to be addressed as processes embedded in society, taking into account existing power imbalances and struggles. We also would like to emphasize that CRAFT explicitly invites journalists to participate in this discussion and share their experiences, insights and stories.

AlgorithmWatch co-founder and ED Matthias Spielkamp joined the selection committee and will serve as a reviewer of contributions. The ACM FAT* conference will take place in Barcelona, Spain, from 27 to 30 January 2020. This is the first time that the conference is held in Europe. The submission deadline for proposals is 12 September 2019.

You can find the call below and at https://fatconference.org/2020/callforcraft.html

CRAFT Call (Panels, Workshops, Debates, Unconferences, ...)

proposal pre-registration: 9 September 2019
proposal submission: 12 September 2019
notification date: 3 October 2019

The ACM FAT* conference has predominantly focused on fairness, accountability, and transparency. The success of the field of fairness, accountability, and transparency has also attracted much critique and renewed attention to the limitations of achieving these goals in systems that implement statistical, machine learning, optimization, or autonomous computing techniques. A number of prominent studies acknowledge that addressing societal problems embedded in such computing systems may require more holistic approaches.

In the spirit of reflection and response, we invite academics of all disciplines and people representing different communities of practice (including journalism, advocacy, organizing, education, art, public authorities) to contribute to a program that will be subject to its own review process and that will be embedded in the ACM FAT* conference (the CRAFT call is very much inspired by the NeurIPS 2018 CRACT workshop.) This call invites contributions to that program in the form of workshops, panels, and other formats to:

1. address critiques of the field of fairness, accountability, and transparency, such as its blind spots,omissions, or alternative possibilities that take a more holistic approach,
2. open future lines of research, collaboration, and practice.

In addition to contributions that explore the problem space in greater depth and from broader perspectives, we particularly encourage proposals that explore solution spaces, indicate mechanisms for positive change, or open possibilities for a greater conversation around countering automated injustices. We value proposals focused on interaction among participants, and we look forward to formats that allow participants to explore starting assumptions, prior experiences, or competing values and to foster community building, shared knowledge production, and future engagement.

Note (Tutorials track): in addition to the CRAFT program, ACM FAT* also solicits proposals on the Tutorials track. Please note that CRAFT emphasizes reflection and critique and differs in important and subtle ways from Tutorials. By offering contributors a space to address blind spots and omissions, offer alternative approaches, and open new possibilities for the field, CRAFT seeks contributions that challenge. By contrast, Tutorials primarily educate and inform. We expect that CRAFT will feature unconventional ideas in engaging and diverse formats. Tutorials, by contrast, might include explainers regarding existing tools, overviews of bodies of literature, or best practices. Some proposals we receive under the CRAFT program may be more appropriate as proposals for Tutorials track, and vice-versa. In such cases, the Tutorial and CRAFT Co-chairs may transfer such proposals to the other program, in consultation with Coordinator(s). In the event that a CRAFT decision suggests your proposal for the Tutorials section of the ACM FAT* 2020 program, you will have one week to confirm whether you accept this choice.

Themes

Below we offer a set of themes, some of which are based on existing critiques, and others which point to new and emergent areas for research. Each theme is accompanied by a non-exhaustive list of questions that a proposal might seek to investigate, as a way of fleshing out the theme.

We think of these themes as “dimensions” rather than “categories”. We invite you to identify the primary theme to which your proposal connects. You are welcome to mention additional themes that are relevant, but please explicitly name the most prominent theme that informs your proposal.

Theme 1: Modeling and (Non-)Deployment

• How do we build the right (economic/social/legal/political) incentives and mechanisms to deploy and evaluate systems with fairness aspirations?
• What are ways to evaluate and monitor the downstream consequences of unfair systems? Are there forms of evaluation that enable those affected in engaging in the evaluation of the damage?
• Are there differences in how we should design algorithms addressing injustices and fairness when these will be used across different contexts (e.g. through APIs) rather than in a single application? How will fairness models and aspirations fare as such systems become ‘legacy’?
• What role can computer science play in developing processes for drawing red-lines that may be applied to current, legacy or future machine learning systems?

Theme 2: Values, Assumptions and Context

• How do we surface hidden values and assumptions embedded in current perspectives on fairness, accountability and transparency, and how do we provide new and alternate perspectives across systems, populations and regions?
• How can FAT* learn from critics who point out that systems affect people in situated ways and that system design requires that designers understand power structures, exploitation, marginalization and other theories of subjugation?

Theme 3: Generating Higher Order Critiques

• How do we turn the FAT* process inwards on our own community and reflect on how, cognizant of power differentials, we might improve research processes internally as well as interactions between and engagement with different disciplines?
• What are the methods and processes from different fields that may come to help in the exploration of inequitable outcomes and mechanisms and how to mitigate these?

Theme 4: Emerging Problems

• What are (a) unresolved questions in existing fields that need to be addressed, (b) desirable operating characteristics for FAT* frameworks in particular application areas that have yet to be explored, or (c) new frontiers of FAT approaches that require rethinking current practices?

Guidelines for CRAFT Contributors

Session Formats for Contributions

In the spirit of openness, we welcome contributions in a variety of different session formats. These may include, but are not limited to the following:

• Interactive Workshop
• Panel
• Debate
• Unconference
• Art Exhibit or Other Artistic Intervention
• Site Visit
• Poster and/or Demo Session
• Lightning Talk or Rump Session

If you would like to propose another format or want to mix and match formats, please get in touch with us. In general, we welcome formats that differ from the conference program and avoid dominating the time with frontal presentations. We especially encourage proposals that bring together people from different backgrounds, be it people of different disciplinary, epistemological, institutional practices, as well as contributions that bring in and bridge different communities, movements, organizations, as well as positions.

Roles and Responsibilities for Contributors

Because we are open to as many formats as possible, we recognize that proposals may contain people who play different roles. Please name these roles in your proposal. For example, some potential roles to mention are:

• Coordinator: The person(s) who are responsible for executing the proposal and contact person(s) for the CRAFT Co-chairs. Prior to the conference, the person(s) may serve as the ringleader for the proposal. During the conference, the coordinator(s) are expected to facilitate discussion, help maintain productive interaction, and encourage participation. We especially encourage coordinators to ensure they have people on their team that can bridge across different communities of practice.
• Presenters: Those who are expected to present at the proposed session.
• Documenter: Because CRAFT is a new space within the ACM FAT* 2020 program, we strongly suggest that you mention someone who will help to document your contribution for posterity’s sake.

Important: Note that you can submit no more than two contributions to CRAFT as a coordinator.

Important: If you are proposing a workshop, panel, poster session or another group format, you will need enough time to either circulate your own call or wrangle people who will participate. Your proposal should include a timeline for distributing a call, as well as for completing your final program.

Proposal Requirements

Your initial proposal should include sufficient information to evaluate it equally among other proposals. We provide a template here so as to streamline the proposal process, as well as the selection process.

• Proposed Title: Let us know what your session will be called. (Max. 25 words)
• Coordinator(s): Please indicate the name or names of session coordinators, including their titles, affiliations, and contact information.
• Presenter(s): If known, indicate the name or names of session presenters, including their titles and affiliations. (Note: The timeline below may be helpful.) If you don’t know these, please say so. Where possible, briefly describe what they will present on or contribute to your session.
• Documenter: If you intend to have someone document your session, please indicate their name, title, and affiliation here.
• Format: Indicate and describe the kind of format you will use for your session.
• Length: Choose the appropriate length for your session. The proposed events can be 1.5h, 3h or 6h in duration. Events that require 6 hours will be split over two days.
• Description: Indicate the problem and/or critique you are engaging and the learning (or otherwise) goals of your session in approximately 800-1000 words (e.g., the form of an extended abstract). What are you trying to achieve with your session? Where do you want participants to be by the end of your event? Please also make sure to indicate how you will know that your participants effectively engaged with the problems/solutions, or critiques, reflections, visions that your session seeks to generate. Please note, the title, coordinators’ names, and a shorter description (max: 500 words) will be listed in the ACM FAT* 2020 program.
• Primary Theme Your Session Fits Into: Tell us which theme (see above) your proposal fits into.
• On-site/Offsite Location: Let us know whether your session will take place within the ACM FAT* 2020 conference space, or whether you intend to run your session off-site. If you intend to do something offsite, please explain whether you have already confirmed another space and by when you expect to have a confirmation, in addition to a description of how the participants will reach the off-site location.
• Target Audience Size: Let us know how many people you anticipate coming to a session or how many participants you would ideally like to have. At the moment for on-site workshops we have two types of meeting rooms: MEDIUM meeting rooms for 50-60 people in a standard setting (seated, looking forward) or 20-25 people in a round-table setting, and a LARGE meeting room that fits 200-300 people
• Documentation and Reporting Plans: We would like to ensure that CRAFT sessions are properly documented. Indicate how you plan to capture and/or catalyze ideas that arise in your session (e.g., mention format and technical infrastructure for documentation during  event). We plan to compile reports of each session (max 1000 words), with the potential of producing a special issue. Please indicate your interest in a special issue. Deadline for reports are March 2020.
• Publicity Plans: If you plan to use social media to promote or document your session, please indicate a hashtag here.
• Other Needs: Include a note if you have any special requirements for your session that are not addressed elsewhere. By default for on-site sessions, audio/video support including projector, a laptop, speakers, podium, podium mic, lapel mic, and handheld mic will be provided. CRAFT sessions will not be recorded by the ACM FAT* 2020 conference organization.

Timeline

• CRAFT call for contributions announced: July 15th
• CRAFT submission link goes live: July 30th
• July-August -- Coordinators solicit participants for their sessions
• CRAFT proposal pre-registration: September 9th
• CRAFT proposal submission: September 12th
• CRAFT Co-chairs assign proposal titles to selection committee: September 9th-12th
• Selection committee submits reviews: September 24th
• Tutorial and CRAFT Co-chairs deliberate: September 25th-30th
• CRAFT notification date: October 3rd
• CRAFT session proposal description (500 words) due: December 5th
• CRAFT program published: December 12th
• ACM FAT* 2020 Conference registration opens: mid-December 2019
• CRAFT sessions: January 29th and/or January 30th, 2020
• CRAFT Coordinators submit final reports: March 2020

Reviewing Proposals and Making Recommendations

Keep in mind that we are looking for proposals that:

• Are accessible to people coming from multiple communities of practice (academic, industry, journalistic, artistic, activist, organizing, etc.)
• Go beyond a conventional academic approach of critiquing FAT*
• Offer a vision that helps fulfill the aims of the CRAFT call

In general, reviewers will look favorably upon proposals that demonstrate that you have thought through the steps needed to achieve the goals of your session. So please be as detailed as possible. A list of reviewers will be made available shortly.

Evaluation Criteria for CRAFT Session Proposals

CRAFT uses a single-blind process for review, which means the Coordinator(s) do not know who the reviewers are, but the reviewers know who the Coordinator(s) are. Each proposal will be evaluated by two people: one who is assigned to a particular thematic area, and one person who is outside the thematic area. To encourage reflexivity and support a range of perspectives and practices in the CRAFT program, our selection committee will reflect reviewers coming from different backgrounds, experiences, and expertise on automated decision systems.

The review process will focus on five basic areas:

1. Organizational details: Is the proposal clear, sensible, and thorough?
2. Rigor and quality: The potential for the topic of the proposal to generate stimulating discussions and useful results.
3. Planning: The coordinators’ ability to demonstrate a well-organized process and plan for a contribution that is welcoming and fosters interactivity.
4. Contribution and relevance: Is the proposal about critique and reflection? Does it address blind spots, omissions, or dominant approaches of FAT* work? Does it consider or offer alternative, holistic approaches? Does the proposal open new lines of inquiry, collaboration, or practice?
5. Catering to the audience: Is there a clear and workable plan for facilitating a lively environment for discussion for all participants?

If multiple submissions are received on the same or similar topics, the organizers may be encouraged to merge them or differentiate them. Ideally, one or more of the organizers should have experience in organizing similar events.

Please note the CRAFT program is not a separate event from the ACM FAT* conference. All participants—whether CRAFT contributors, Tutorial presenters, Paper presenters, or other attendees—will benefit from participating in the entire conference program.

Roles and Responsibilities of CRAFT Co-chairs

As Co-Chairs of CRAFT, we will work with a selection committee (i.e., a set of reviewers) to guarantee a fair decision making process as well as a diligent process for participating in CRAFT. This includes due process in selection, timeliness, openness to critique and respect for others. Finally, we are responsible for making a final program in which all CRAFT activities are listed and made available to conference participants.

As Co-Chairs, we are committed to upholding the following throughout CRAFT:

• We respect the dignity, experiences, and perspectives of the communities impacted by the systems that are under discussion at CRAFT.
• We recognize the problematic effects of systemic optimization but also that not all populations and communities are impacted equally. In fact, some are likely to benefit economically, socially, or otherwise, while others are suffering from these systems.
• We respect the dignity, experiences, and perspectives of each CRAFT participant.
• We recognize multiple forms of evidence for the way socio-technical systems operate and for their effects.
• We are cognizant of the role power plays in shaping how we inhabit rooms, and speak up and shape urgencies or commit to challenge unjust existing power structures such as (but not limited to): sexism, racism, and transphobia.

CRAFT Co-Chairs

Please contact craft@fatconference.org for any questions.

• Seda Guerses, TU Delft
• Seeta Peña Gangadharan, London School of Economics
• Suresh Venkatasubramanian, U. Utah