The NESTOR project is extremely ambitious, as it sets out to create “an entirely functional, next-generation, comprehensive border surveillance system offering pre-frontier situational awareness beyond sea and land borders”.
Deliverable D2.2 specifies that, according to EU Regulation, the concept of “pre-frontier situational awareness” refers to “the geographical area beyond the external borders which is relevant for managing the external borders through risk analysis and situational awareness” — which basically expands border surveillance beyond border areas.
This is all the more important as the project conflates immigration and criminality in the description on Cordis, arguing that “Europe’s borders are under significant pressure by migration flows, armed conflicts in surrounding territories, smuggling of goods and humans and transnational crime. However, monitoring the routes used by criminal networks is prevented by geographical challenges, such as dense forests, high mountains, rough lands, sea and river areas.”
Allegedly, this situation could only be handled by a system that is “based on the concept of the European integrated border management and relies on optical, thermal imaging and radio frequency spectrum analysis technologies fed by an interoperable sensors network.”
Web, “dark web,” and social media monitoring is also featured, together with “mixed reality” applications and AI for swarms of unmanned vehicles, as — according to the project’s Grant Agreement, which we managed to obtain in a severely redacted form after an access to information request to the Research Executive Agency — “complete mission awareness” is requested.
A 20-mins video presentation is available on the project’s YouTube channel.
NESTOR features several technological components that are broadly described on the project’s Cordis page: “NESTOR long-range and wide area surveillance capabilities for detection, recognition classification and tracking of moving targets (e.g. persons, vessels, vehicles, drones etc.) is based on optical, thermal imaging and Radio Frequency (RF) spectrum analysis technologies fed by an interoperable sensors network including stationary installations and mobile manned or unmanned vehicles (aerial, ground, water, underwater) capable of functioning both as standalone, tethered and in swarms. NESTOR BC3i system will fuse in real-time border surveillance data combined with web and social media information, creating and sharing a pre-frontier intelligent picture to local, regional and national command centers in AR environment (augmented reality, ed.) being interoperable with CISE and EUROSUR.”
Deliverable D7.4 offers a more detailed description: “To achieve its objective, a number of technologies are employed and integrated into a standalone platform that can easily be deployed for surveillance of the areas of interest.” Outputs include the “NESTOR Border Command, Control, and Coordination intelligence (BC3i) system,” which “uses cutting-edge Artificial Intelligence and Risk Assessment technologies, fuses in real-time the surveillance data in combination with analysis of web and social media data, sharing a pre-frontier intelligent picture to local, regional and national command center operators in an AR environment.”
The deliverable adds that “surveillance data are collected from several sensors which are deployed in the area of interest, both land and maritime environments. The collected data are transported through a secure communications network and are fused, correlated, and processed to extract surveillance-related information.”
Social networks are also to be “monitored for enriching the information base and enhancing the level of situational awareness and the decision support capabilities of the platform. BC3i (Border Command, Control and Coordination intelligence) is the core component that combines and presents the collected and fused information to both the Control center and also to field LEAs, while presentation may be enhanced though AR technologies that will be integrated into the platform.”
From the (severely redacted) Grant Agreement we managed to obtain, we learn that the “web and social media monitoring” task means that data acquired from the web and social media (even from the dark and deep web) “will aim to identify content related to illegal border activities including suspicious adverts, groups, sale of goods, potential trafficking routes as well as information posted by bystanders.”
The “Raw web and social media data” thus obtained “will be transformed to an entity-link model to allow Natural Language Processing, Social Network Analysis and Indicators to be relations to specific scenarios, such as drug trafficking or illegal activities and a rule-based expert system used to determine the influence each indicator has on the outcome of the analysis. The task will aim to support reverse geocoding, entity, date and other keyword extraction, indicator detection as well as analysis of networks and trends.”
All deliverables related to this Working Package are however labeled “confidential,” to be read only by “members of the consortium (including the Commission Services).”
The Grant Agreement states that objectives of another Working Package include the development of
1) “mixed reality functionalities” able to “consume and visualise in near-real time data and video streams”;
2) “collective intelligence for swarm robotics” (more specifically “an AI-based service for autonomous designing of missions for a swarm of UxVs to cooperatively cover large off-shore areas”;
3) “multi-modal data fusion,” i.e., “the necessary algorithms to process and fuse detection information from multiple sources and from different services”.
Deep learning algorithms are also to be developed, to “analyse the acquired video streams.”
Deliverable D7.6 (“Standardisation and collaboration with other projects,” obtained by AlgorithmWatch through an access to information request) depicts the NESTOR project as “a basis for future EU-funded projects to continue the work started by NESTOR,” and describes the decision to “carry out an action to create a preliminary standardisation roadmap for the border management domain.” 9 other EU-funded projects joined the action group: AI-ARC, BORDERUAS, EFFECTOR, ENTRANCE, ISOLA, MEDEA, PROMENADE, ROBORDER, and METICOS. FRONTEX was a “supporter.”
In August 2022, NESTOR joined the BES (Border External Security) Cluster of EU-funded projects that includes, among others, ROBORDER, iMARS, and D4FLY.
More specific “synergies” have been developed with EFFECTOR, PROMENADE, BORDERUAS, MEDEA, MELCHIOR, COURAGEOUS (which “will develop a standardized test methodology for detection, tracking and identification of nefarious drones’ utilising countermeasure systems to protect the lower airspace,” and might therefore be related to NESTOR’s anti-nefarious drone technology, ed.), and FLEXI-CROSS (the rationale of which is briefly explained on a project newsletter).
All deliverables related to pilots/trials (WP6) are either “confidential” or “classified,” so most details were not available to the public at the time of writing. We do however have some information concerning project pilots.
Three international land and maritime border trials have taken place to assess the NESTOR platform: 1) Lithuanian Maritime Trial (14-18 November 2022); 2) Cypriot Maritime Trial (30 January – 3 February 2022), involving a SAR incident; 3) Greek-Bulgarian Maritime Trial (12-22 March 2022), involving an incident of cross-border human smuggling; and an analysis of social media posts on human smuggling through a web-crawler. Identified smugglers are followed by multiple mobile and fixed cameras, UAVs and UGVs, sensors, all coordinated through “mixed reality glasses” and a mobile device app, informing “ground force intervention” against “illegal migrants.” Real-time tracking of vessels and neutralization of hostile drones (by an “anti-drone device”) were also featured.
More details are included in Newsletter #4 in which the trials are declared “a resounding success.”
Importantly, a Border Monitoring Violence Network report on the Evros region in Greece alleges that NESTOR outputs — similarly to ANDROMEDA’s — could have outlasted the pilot phase to quietly be deployed altogether: “Visual material collected during the field assessment of the area confirms that the NESTOR equipment is still in place, suggesting the possibility that it is used by the Hellenic Police”.
Even though the project ended on April 2023, only five deliverables are publicly accessible on the project’s Cordis page 18 months later. Thanks to information disclosed by the REA in response to a request for information access by AlgorithmWatch, we learned that the project consists of a total of 88 deliverables, each of a length that “varies widely, from 10 to around 380 pages.”
Almost half of them — 40, to be precise — are listed as “non public.”
A list of project deliverables published in the Grant Agreement we obtained reveals that the large majority of deliverables is either “confidential” or “classified.” For example, all technical requirements (WP5) are “confidential” and all deliverables related to trial (WP6) are either “confidential” or “classified.” All deliverables related to ethics (all deliverables in WP8, plus D1.5 and D1.6) are “confidential.”
These deliverables deal with research ethics (in recruiting participants, obtaining their consent, etc., D8.1), approval of an ethics committee for research on humans (D8.2), data protection for subjects involved in the research (D8.3), health and safety for staff and research participants (D8.5), dual use and its implications (D8.6), and prevention of the misuse of research findings (D8.7).
D8.4 concerns “Detailed information to demonstrate that fair benefit-sharing arrangements with stakeholders from low and lower-middle income countries are ensured,” “Details on the materials (including personal data) which will be imported to/exported from the EU,” and “If relevant, copies of import/export authorisations, as required by national/EU legislation.”
D8.8 argues: “Due to the severity of the ethics issued raised by the proposed research, a report by the Ethics Board must be submitted as a deliverable at M12.”
While marketed as capable of saving lives, all of this is tailored to the needs of the border security community. The Grant Agreement states explicitly: “Each technology objective concerns a specific challenge defined by the border authorities’ current and foreseen needs.” This is hardly surprising, as NESTOR is coordinated by the Hellenic Police, and built under a “crimmigration” assumption (i.e., one that fundamentally ties immigration and criminality, ed.).
Several issues were raised during in the “Ethics Summary Report.” For example: “The ethics issues concerning data management including personal data are addressed in the project in a general way.” “Further details on the procedures to address the ethics issues” were “missing.”
Belarus was featured as a non-EU country involved in project activities, and was consequently included among the “research related activities undertaken in these countries” that “raise potential ethics issues.” “Personal data may be transferred to non-EU countries,” including to Belarus thatlater abandoned the consortium.
Several harms and misuses were identified, for example that the research can “involve the use of elements that may cause harm to the environment, to animals or plants,” and that it can “have the potential for military application” (tellingly, the first paragraph of the comments on this “dual use” feature is redacted).
The report notes that research for NESTOR has “the potential for malevolent/criminal/terrorist abuse”: “The project aims at development of the surveillance system, capable of collecting and processing vast amount of personal data and tracking of individuals, which has a high potential for misuse.”
The Ethics Summary Report says that while “the applicants recognise this potential and declare in Section 5 of the proposal that measures to mitigate the risk of misuse will be taken,” “no details are given or indication on which project deliverables will address the issue.” The document only recommends “to consult and adhere to the principles of the Ethics Guidance on Trustworthy AI” as a response.
The “Ethics Opinion” that was formulated as a result only grants a “Conditional ethics clearance” to the project, which means that “clearance is subject to conditions, ie ethics requirements. The requirement must either be fulfilled before grant signature or become part of the grant agreement.”
Among the “Post-Grant Requirements,” it was consequently required of the beneficiary to submit 1) a “clarification of the measures to protect vulnerable individuals/groups and minimise the risk of their stigmatization,” 2) “details on potential dual use implications of the project and risk-mitigation strategies,” and 3) “Risk assessment and details on measures to prevent misuse of research findings.”
By the end of the project, however, deliverable D1.6 —Ethics and societal issues management final report — claims that all ethical issues have been “analysed and described in detail in the respective WP8 deliverables, as well as the mitigation measures to effectively counter the possible risks” previously identified. Two DPIAs were conducted. Full compliance with fundamental rights, international law, and EU law (and ethics principles) is declared (D8.2).
This also concerns potential harms from AI: “Albeit not included in the WP8 ethics requirements, a questionnaire on ‘Ethics by Design for AI’ was completed by the NESTOR technical partners that designed and used AI systems during the NESTOR research,” notes Appendix G in section 2.2.
Contrarily to other projects that employ AI in border surveillance and management, NESTOR is portrayed as beyond the scope of the AI Act. Project member CERTH were following a “human-centred approach,” respecting “human agency” as “it does not suggest actions to the operators” and “does not perform any decision-making.” Therefore “the AI system does not autonomously make decisions about issues that are normally decided by humans.”
More risks arising from the adoption of AI in border controls and management are detailed in pp. 27-28: “Whereas great attention has been paid to the issue of bias and discrimination, it must be noted that even accurate and unbiased AI systems may pose significant other risks, including to data protection and privacy. (…) Moreover, even when profiling is not based on biometric or personal data (such as in the case of NESTOR, as the Consortium alleges), other types of data or combination thereof used for algorithmic profiling may lead to discrimination based on prohibited grounds.”
It should be explored whether it would actually be possible “to perform online and social media acquisition in a targeted manner that limits the opportunity for accessing such data and/or making use of pseudonymisation or anonymisation techniques to further safeguard any potential processing of such data.” After all, deliverable D2.2 states that ”issues surrounding the legality and ethics of data collection from the web are still a grey area.”
D1.6 concludes that “NESTOR has been carefully designed to not only respect but also promote fundamental rights,” and that, “while a few points of caution are traced with regards to possible extensive use of this technology in the future, there is no doubt that end users, currently, would be multiply benefitted and that the NESTOR system will constitute contribution to society.”
