AlgorithmWatch documented the adoption of automated decision-making (ADM) systems to determine COVID-19 vaccination priority orders both by the Italian Federation of general practitioners and by regional authorities in Lombardy, Valle d’Aosta, and Piedmont. More regions could be about to follow their lead. Both the proponents and the developers of algorithmic prioritization claim that dialogue with other local administrations in the country is ongoing.
However, our analyses cautions that the workings of these algorithmic solutions are not always transparent — and that, even when details are provided, methodological issues remain, which could potentially lead to discriminatory results.
While the ADM systems we analyzed differ, they all share an underlying rationale: maximizing efficiency in vaccine distribution while prioritizing older people and those who are more at risk because of comorbidities – where a patient suffers from more than one health condition – and the individual’s overall medical history.
This has been a hotly debated topic in Italy recently. Even during rollout, several factors changed the prioritization criteria. Firstly, a new government came to power in the middle of the pandemic, together with a new COVID-19 Special Commissioner. Secondly, a very small number of people who had received the AstraZeneca shot suffered from a rare blood clot. This led to major strategic shifts. The AstraZeneca vaccine – which had at first been suggested for those aged 18-55 - was temporarily suspended, before being recommended again (for the over 60s only). As things stand, the AstraZeneca vaccine is being re-considered for use in all age groups. When it became known that thousands of people had somehow managed to jump the vaccine queue and receive shots they were not yet entitled to, a heated State versus regions debate raged.
On 24 March, in a speech at the Senate, Prime Minister Mario Draghi denounced “major regional differences” related to vaccination coverage of those over 80. “These are difficult to accept,” he added, arguing that some regions were willingly “neglecting their elderly in favor of groups that proclaim priority, probably on the basis of some contractual influence.”
“With what conscience does someone jump the line knowing that they are leaving a person who is over 75 or fragile exposed to the real risk of dying?” Draghi later added at a press conference, before going on to say: “Stop vaccinating people under 60.”
This resulted in an ordinance on 9 April, by COVID-19 Commissioner Francesco Paolo Figliuolo, that prioritized the vaccines for those aged over 80, followed by severely fragile individuals (i.e. those suffering from a number of conditions listed by the government) and those aged 70-79, followed by individuals in the 60-69 age bracket. For people like teachers and school personnel - who could previously get the vaccine regardless of age of health status – they would now have to wait for their turn instead.
However, conflicts between the prime minister and the regions continued, when some regions decided autonomously to give vaccinations to individuals under 60 (for example, Lazio and Calabria) or to categories of people (such as tourism operators in Campania) that were not supposed to be prioritized.
No algorithm can resolve such institutional conflicts. And yet, many saw an opportunity for algorithms to provide a more nuanced, informed and evidence-based assessment on how to comply with the vaccine distribution criteria identified by the government
Valle D’Aosta: a machine learning model that evolves with the pandemic
The Valle D’Aosta region, one of the richest in the country, plays a unique role among the forerunners of algorithmic vaccine prioritization. The administration promoted the idea of using sophisticated machine learning algorithms to categorize the population by age and a series of variables related to each individual’s overall well-being. The administration compared different algorithmic prioritization solutions, evaluated the consistency of their results, and ultimately picked the most conservative ones — i.e. the algorithmic choices that resulted in the urgent protection of those at greatest risk from severe forms of COVID-19.
This is the result of a data-driven approach used in this autonomous region even before the pandemic. Valle D’Aosta authorities first commissioned a study by technology provider GPI to obtain a risk-based categorization for chronic diseases within the population. “We initially investigated diabetes, chronic obstructive pulmonary disease, hypertension and heart failure,” says Antonio Colangelo, Head of Research and Development at GPI in a call with AlgorithmWatch. Using information from NSIS (Nuovo sistema informativo sanitario), including “granular data” on — among other topics — hospitalizations, access to emergency rooms and the distribution of medicine to hospitals, researchers derived a “prevalence indicator.” This allows authorities to know “how many individuals suffer from a certain pathology in a certain municipality at this very moment,” Mr. Colangelo said.
However, some citizens might not want to be included in certain public health databases, to keep information on certain diseases confidential. Also, many suffering from chronic disease might not have been hospitalized over the last five years — the period covered in the study. For those reasons, GPI decided to use ”complex algorithms” that can indirectly estimate how many people actually suffer from a disease by adding the ATC codes (codes that identify each medication) into the mix, thus taking into account the kind of medication a patient took over the years. According to Mr. Colangelo, linking drug consumption to a condition is not a straightforward exercise and one that needs to consider more nuanced factors instead. For example, data around resources “absorbed” by a patient — hospitalizations required, medical exams taken, etc. — all need to be included as well, when categorizing for risks and chronicity.
However, once this is done, it is much easier to come up with a rigorous, algorithm-driven order of priority for COVID-19 vaccination. “It only takes adding some Boolean variables, indicating whether a patient suffers from one of the pathologies indicated by the government,” Mr. Colangelo said. This process enables the authorities to categorize the population into four risk clusters. They can go beyond simply using the criterion of age to come up with an informed, sophisticated notion of vulnerability instead.
The calculations GPI derived in this way predict that those in cluster 0 (deemed low risk) will see their vaccination delayed by 10 days compared to those in higher priority groups based solely on age. At the same time, and more importantly, people belonging to cluster 3 (deemed to be at greater risk of developing severe COVID-19) will be vaccinated 61 days before an age-based prioritization order. This process could have a dramatic impact on hospitalizations and death rates.
This is not a mere study. Mr. Colangelo describes it as an operative Machine Learning model, one that is capable of autonomously adapting the shape of the cluster according to fresh data. Out of this, another fundamental insight can be derived: the fact that priority should be, in many instances, given to individuals belonging to younger age cohorts. In fact, of the around 14,500 over-80-year-olds in Valle D’Aosta, GPI inserted 575 in cluster 3; at the same time, more than 700 individuals aged 55 to 69 years old also qualified for the same high-risk cluster. This means that adopting a prioritization that assumes age alone is the most important criterion most certainly fails to include several younger high-risk individuals among those who should be vaccinated first.
It also means that the criteria adopted by the Draghi government are not optimal. “Cluster 3 is much denser for those aged 45 to 70 than for those over 80,” Mr. Colangelo said. “We are getting our priorities wrong.”
Valle D’Aosta, which also deployed Artificial Intelligence through a virtual assistant and chatbot to help authorities contact and schedule vaccination appointments, aims to vaccinate 75 percent of the population by the end of July. Its solution is also currently being discussed with “many” other institutional actors in the rest of Italy, Mr. Colangelo added.
AlgorithmWatch requested access to GPI’s technical documentation. We were particularly interested in the methodological aspects related to the algorithm’s validation. However, Mr. Colangelo said that the material is not publicly accessible, as it has been “exclusively” provided to Valle D’Aosta authorities.
“This shouldn’t be acceptable,” argues Ciro Cattuto, an associate professor who is working on infectious disease dynamics and digital epidemiology at the computer science department of the University of Torino. “Especially at a time when a great deal of sacrifice is being asked of citizens, transparency should be a duty.”
Lombardy’s “Covid Vulnerability Score”
Valle D’Aosta authorities also had the chance to consider another algorithmic solution for COVID-19 vaccine prioritization: the “Covid Vulnerability Score” made by Bicocca University researchers within the “StrESS” project, an academic study. This is another attempt at clustering COVID-19 risk categories, and Lombardy adopted it in its own vaccination plan.
In this model, the vulnerability index was obtained via the analysis of data about 16 million citizens from five regions (apart from Valle D’Aosta and Lombardy, Marche, Puglia, and Sicily were also included). The model crossed health data contained in the Banca Dati Assistiti (a regional database of patient data) with data cataloguing tests, hospitalizations and deaths in the same regions during the first and second wave of COVID-19.
The system, which was proposed and developed pro bono by Bicocca University professor Giovanni Corrao with the help of colleagues, operates under the assumption that: in order to understand how to proceed in the future, health authorities first need to investigate each patient’s past — and that algorithms can do it better. Developers claim that the vulnerability score “allows the prevention of hundreds of intubations and deaths.”
But how is the score obtained? “We first reconstruct the medical case of each individual aged 18 to 79 through algorithms,” Mr. Corrao explained in a call with AlgorithmWatch. “That checks how patients interacted with the health system up until the pandemic.” This results in a clinical profile for each subject. The algorithm then looks for those which most likely correlate with severe COVID-19 infections. The score is meant to precisely measure the risk that each individual has of developing forms of the disease that require hospitalization or that may lead to death.
The resulting “Covid Vulnerability Score” identifies 23 conditions and pathologies that are relevant in assessing such risk — including some that, to Mr. Corrao, have been overlooked by the Italian government. These include neurological diseases such as epilepsy and Parkinson’s, mental health issues and, among others, anemia and lupus.
According to Mr. Corrao, the real issue with how the government defines “vulnerability” is that it lacks analytical detail and rigor. On the one hand, conditions listed in official documents fail to make a distinction between, for example, “cardiopathy in a subject who has suffered from hypertension after an internal organ has been damaged, and in one who recently suffered from a stroke.” On the other hand, “what matters is not a single condition, but the overall clinical profile of a patient.” Mr. Corrao’s algorithm is intended as a solution to provide a much more fine-grained “vulnerability” score, which could then be interfaced with age to provide the optimal prioritization scheme for the vaccine rollout.
But would Mr. Corrao’s algorithm provide a truly “optimal” outcome? According to Mr. Cattuto, the University of Torino’s computational epidemiologist, even though the methodology applied in the Bicocca experiment relies on a basic logistic regression, it constitutes a best practice thanks solely to its simplicity. This allowed researchers to provide both a blueprint for further experiments, and very accurate performance. However, some issues remain.
Mr. Cattuto for example notes the absence of a “spatial dimension” that would allow the inclusion of differences in “socio-economic” factors in the vulnerability score, thus depriving the analysis of important information about how individuals on a low-income are disproportionately affected by COVID-19. “Overlooking the differences that arise from living in rural rather than urban areas, or in suburbs rather than city centers, may result in discrimination,” explains Mr. Cattuto, “as this spatial dimension is a proxy for socio-economic conditions, and these are correlated to health.”
Also, a system like the one adopted in Lombardy only applies to those who are registered with the public health system. “What about irregular immigrants, then?” asks Mr. Cattuto.
Finally, the results obtained through the algorithm are not that different from those that prioritize solely by age, he argues. And indeed, data provided by Mr. Corrao show that the expected “benefit” of adopting algorithmic prioritization amounts to a reduction of around 100 “fatal outcomes.”
Of course, no life that can be saved is negligible. And yet, argues Mr. Cattuto, we shouldn’t be obsessing over obtaining the best possible prioritization algorithm; rather, data provided in the Bicocca experiment show that all of the prioritization strategies that have been considered — even based on age only — work when deployed quickly enough. “Our main problem is not optimization, but rather ramping up vaccinations,” he says.
Mr. Corrao’s project has been presented to officials at the Health Ministry, the Italian Medicines Agency (AIFA), and the Higher Health Institute (Istituto Superiore di Sanità). However, so far, only the institution in Lombardy has adopted it.
Apart from Valle D’Aosta, as previously mentioned. There, authorities compared the GPI algorithm with Mr. Corrao’s and found that their outputs were mostly consistent (“at around 81 to 82 percent,” Mr. Colangelo said).
Piedmont: an algorithm decides whom to vaccinate, when, and even where
It would be interesting to ascertain whether priority lists obtained through a third algorithm, deployed in Piedmont, are also consistent with these findings. AlgorithmWatch managed to get confirmation of the adoption of “a system to decide whom to vaccinate, when and even where,” as the region’s Health Assessor Luigi Icardi described it in a phone call.
The algorithm was developed by the Piedmont Consortium for Information Systems (CSI). However, no detail about how it functions — or even its existence — is publicly available, except for the fact that its categorization includes “a subject’s pathologies.” When asked to provide those details, Mr. Icardi told AlgorithmWatch to contact CSI, the region’s technology partner. CSI refused to share any details, citing a lack of authorization from regional authorities. However, we do know that the system is supposed to send each individual to the closest vaccination center — and that this did not always work as intended.
Local media reported that a group of over 80-year-olds from Vallo Torinese and Traves were assigned to the wrong center by the system, forcing them to travel 50 kilometers to get their jab. Some of the group could not reach the assigned vaccination location and had to turn down the opportunity. AlgorithmWatch contacted Marco Bussone, the president of UNCEM, (Unione Nazionale Comuni Comunità Enti Montani – an organization that represents the interests of mountain municipalities) who lives in Vallo Torinese and publicly denounced the mismatch. He says that health authorities must have rapidly intervened, as no more complaints have been reported in the area.
An algorithm for all general practitioners
Apart from these local examples, an additional prioritization algorithm has been developed for use by general practitioners throughout the country. Promoted by the Italian federation, FIMMG (Federazione Italiana Medici di Medicina Generale), it is part of a freely available platform which is open to all doctors and health institutions. The algorithm prioritizes vaccine appointments according to a series of parameters — including, again, age and vulnerability — and citizens use a digital tool to book an appointment.
Vulnerable individuals are categorized according to data derived from the databases that are available to general practitioners. The stratification of patients is obtained using software developed by NetMedicaItalia in cooperation with Cittadinanzattiva, the department for Engineering of Information at the Università Politecnica delle Marche and a panel of highly qualified experts.
But is this solution actually used?
“Yes, this is not a theoretical exercise. The Machine Learning algorithm has already been developed and it is currently being used by general practitioners all over Italy,” claimed Paolo Misericordia, Head of ICT at FIMMG, in a phone call with AlgorithmWatch. The Federation has 12,000 members, he said, and the algorithm is 95 percent accurate when it comes to identifying “the most vulnerable.” Media reports also indicate that regions such as Basilicata, Marche, Emilia-Romagna and Veneto “have started to reason with family doctors on the models to be adopted.” Lombardy, where the Bicocca algorithm has been implemented, have closed a “preliminary agreement” to use it.
All of which begs the question: what to make of this plethora of prioritization algorithms?
To Mr. Misericordia, the fact that different algorithmic priority orders, derived from different solutions, might be used in the same region (or even on the same subject) is not an issue. When asked whether he would welcome a unified ranking, Mr. Misericordia said that as long as common goals are enshrined in the national vaccination plan, having automated decision-making systems that respond to different databases and organizational models could actually be a plus. “Many subjects are involved, and there’s room for everyone,” he said. “After all, it is not the algorithm that makes the difference, but the databases on which it is applied.”
When it comes to considering a potential nationwide deployment, Mr. Cattuto from Torino University argues that “it would be ideal to have a portfolio of algorithmic solutions,” to then be able to scrutinize each of them in their own context, and evaluate pros and cons analytically. At the same time, we shouldn’t be putting too much emphasis on the technology, he says. “If the debate about digital contact tracing apps taught us anything, it’s that logistic choices are even more fundamental.”
Italy is not the only country in which vaccine prioritization algorithms are being adopted. For example, the “QCovid” algorithm deployed in the United Kingdom, led to claims of inflated risk scores due to missing data and more general doubts about its reliability in the scientific community. Similar issues have also been highlighted in Bavaria, Germany and in the USA.