Spanish National Police Halts Veripol, Its Flagship AI To Detect False Reports

This story was originally published on Civio. AlgorithmWatch edited it according to its styleguide.

“It is the first tool of its kind in the whole world,” the National Police wrote in 2018. Back then, they presented Veripol as an algorithm capable of detecting false reports of violent robbery – with an accuracy of over 90%. In October 2024, six years later, the National Police stopped using this AI-based tool, the Technical Office of the Directorate General of Police told Civio. The Ministry of the Interior stated that the reason for rejecting its use was its lack of validity in judicial proceedings.

The dismissal of Veripol takes place three months after the publication of the European Union Artificial Intelligence Act text in the Spanish Official State Gazette (BOE in Spanish). The AI Act classifies polygraph-like systems as high-risk, they are therefore subject to higher transparency, review, and update requirements than less risky systems. In late September 2024, a group of mathematics and legal experts at the University of Valencia had pointed out serious shortcomings of the tool in a report, first of all the lack of information on how it works.

Veripol was developed by researchers from Universidad Complutense de Madrid as well as Universidad Carlos III de Madrid, in collaboration with police officer Miguel Camacho-Collados, who is currently the head of the Technological Innovation and Cybersecurity department at the State Council. According to the academic article detailing its development, the team trained the tool on a sample of 1,122 theft reports in Spain from 2015, 534 being true and the remaining 588 being false. The texts were simplified and automatically processed using Natural Language Processing (NLP) techniques. The words in the reports were categorized, words that appeared in less than 1% of the sample or in more than the 99% were discarded.

The researchers then applied various statistical regression methods to identify the words most common in both true and false reports. For example, if a report contains the words “day,” “lawyer,” “insurance,” or “back,” it is more likely to be false, according to Veripol's predictions – and is even more likely to be false if the report contains the words “two hundred” several times or adverbs such as “barely.” On the contrary, reports that refer to buses, a particular phone brand, or a car number plate are more likely to be true.

In June 2017, the team behind Veripol conducted a pilot test at police stations in Malaga and Murcia. According to the academic article, the complainant confessed to be lying in 83.54% of the reports identified as false by Veripol. In December of that same year, the Spanish Police Foundation awarded the research, and in 2018 the Ministry of Interior announced its implementation in all police stations.

According to research by AlgorithmWatch, the National Police used Veripol to analyze around 84,000 complaints between 2018 and October 2020. Out of 49,702 complaints the police analysed with Veripol in 2019, 2,338 were marked false using a combination of Veripol's predictions and other means. In 2022, the police used it significantly less often – the University of Valencia report states that the number of complaints the police analyzed with Veripol decreased to 3,762. The police determined 511 to be false.

The same report points out serious deficiencies of the tool, especially the research starting point, being the assumption that 57% of the violent robbery reports at hand are false. The number refers to the high number of unsolved robbery cases. The report also criticizes the small sample the system was trained on: just over a thousand complaints, compared to approximately 60,000 cases of violent robbery that are registered every year in Spain, according to the Crime Statistics Portal. It also notes a lack of information as well as a lack of a standardized training protocol for police officers. La Voz de Galicia reported in 2020 that the police station in Vigo (one of the main cities in the northern region of Spain), which had been using the program since 2018, had to end it due to the poor training of its officers.

In addition, the complaints analyzed by Veripol were actually written by police officers, so it is not a word-by-word adaptation of the complainants' statements: “It does not analyse the story that the potential liar is telling the police, but analyses the story that the police officer writes himself," the University of Valencia report states. The tool also fails at identifying language differences across Spain.

“The system is not transparent,” the University of Valencia researchers write: “There is no official data available on Veripol at all.” In February 2023 and again in December 2024, Civio requested information on the technical functioning of Veripol and its use, but to date the Ministry of Interior has not even reported the number of police stations that used it.

Information requests

In February 2023, Civio requested the technical specifications of Veripol from the Ministry of Interior, its use cases, and any other documentation on how the application works and what information it contains or may contain. Faced with the refusal of the Ministry of Interior to provide this information, Civio filed a complaint under the Transparency Law to the Council for Transparency and Good Governance, which on 31 October of the same year ruled in favor of Civio and ordered the Ministry of Interior to provide the requested information. The only then provided information were links to press releases published by the Police and the Complutense University on its implementation and the prize awarded by the Spanish Police Foundation.

In December 2024, Civio again requested information related to Veripol, specifically the list of police stations that had implemented the tool as well as their usage data, including the number of cases processed per year and the percentage of cases in which the tool had concluded that the complaint was false. The Ministry responded by stating that they had stopped using Veripol on 21 October 2024 and refused to provide usage data. Civio filed another complaint to the Council for Transparency and Good Governance to access this data, which is still pending.

About the game

David Cabo and Ana Villota participated in the genesis and implementation of the game.

In a pre-print of an article published in the journal Knowledge-Based Systems, called “Applying automatic text-based detection of deceptive language to police reports: Extracting behavioural patterns from a multi-step classification model to understand how we lie to the police” (Lara Quijano-Sánchez, Federico Liberatore, Jose Camacho Collados and Miguel Camacho-Collados, Cardiff University, 2018), Civio found 2015 data on the tool, as well as a list of 110 terms, translated from Spanish to English. The authors were contacted, but they have not provided the original list of words in Spanish.

To avoid ambiguities arising from translation, Civio included only those words in the game that are explained in context. The frequency of specific words determines if a report is considered probably false. This is the list of such terms with their respective weightings (the higher the value, the greater the influence): day (0.48), lawyer (0.43), insurance (0.24), back (7.74), backpack (0.10), shoulder (17.99), helmet (26.92), iPhone (25.56), Apple (0.23), barely (0.19), behind (0.12), two hundred (0.30), euros (6.81), cash (19.18), contract (19.19).

Civio selected for probably true cases the following words: bus (0.52), number plate (0.19), chain (16.06), police (0.31), Chinese (52.58), neck (16.62), portal (0.26), landing (0.36), even (72.31), beard (0.34), centimetre (0.09), thin (0.09), dark (0.10), shout (40.67), grab (0.12), doctor (16.75), friend (0.13).

The visualization of the game was developed with Svelte.js.