Amnesty requires ban on AI risk-profiling methods
The usage of synthetic intelligence (AI) for predictive profiling or danger evaluation ought to be banned in a spread of “high-stakes contexts”, reminiscent of policing, migration and welfare, says Amnesty Worldwide, on the premise that it may possibly solely entrench present patterns of discrimination.
In accordance with a report revealed by the group on 11 June 2026, automated AI-powered danger profiling methods are resulting in false legal accusations in opposition to marginalised teams, and are incompatible with worldwide human rights regulation.
The report highlights how AI danger profiling – utilized by public authorities in regulation enforcement, social safety and migration to determine potential offenders and assess whether or not an individual or group is more likely to break a regulation, earlier than an offence is dedicated – is leading to discrimination primarily based on race, ethnicity, socioeconomic standing and incapacity, violating the precise to equality and non-discrimination.
“The best way these instruments are deployed might be the results of pre-existing stereotypes and prejudices which regard marginalised teams as inherently legal or harmful,” mentioned Alexander Laufer, Amnesty Worldwide Netherlands researcher on expertise and human rights. “People or teams are remodeled from statistical, hypothetical suspects into precise suspects, solidifying pre-existing prejudices or producing new ones. That is the results of present systemic discrimination.”
AI-based danger profiling methods educated on discriminatory knowledge
Thus far, the expertise has been deployed throughout Sweden, Denmark, France, the Netherlands and Australia, in social fraud investigations and debt restoration schemes.
Use of 1 AI system was suspended in Sweden in November 2025, after investigations discovered that the expertise disproportionally and incorrectly flagged marginalised teams for investigation over social advantages fraud.
Previous to the suspension of this method, Amnesty described using this profiling system as “akin to a witch hunt”.
Information primarily based on present systemic discrimination means marginalised teams might be focused with an elevated danger of being labelled as “suspicious”. Amnesty flagged that this contains “racialised individuals, Muslims, individuals on the transfer, individuals with disabilities or continual sicknesses, and other people on low incomes”.
Laufer mentioned AI danger profiling can result in false legal accusations, imprisonment, homelessness, deportations, and denial of social advantages, whereas the shortage of transparency within the expertise can depart people “unable to problem these methods and choices that have an effect on their rights”.
He added: “It additionally poses dangers to different human rights, reminiscent of the precise to honest trial, the presumption of innocence, the precise to privateness and knowledge safety, social safety and an ample lifestyle, and full realisation of human dignity.”
Can AI algorithms predict potential legal behaviour?
Regardless of issues raised by Amnesty and others that the expertise is persistently inaccurate and scientifically doubtful, governments proceed ramping up their use of AI-powered danger profiling as a “cost-effective strategy to combat crime, social safety fraud and irregular migration”.
Nevertheless, in Laufer’s view, it’s “unimaginable to design an goal or impartial danger profiling algorithm – knowledge about individuals is rarely goal”.
The report added that the info required to foretell whether or not a person will commit against the law doesn’t and can’t exist, that means unreliable and biased proxy knowledge is getting used as indicators to foretell potential legal behaviour.
It additionally famous that, as a result of human behaviour is “inherently indeterminable” and “adaptive”, it’s unimaginable to foretell future behaviour precisely.
“In some circumstances, there isn’t a believable connection between observable knowledge and the proposed behaviour being predicted, reminiscent of between race or ethnicity and criminality – particularly, racial profiling,” it mentioned.
“In different circumstances, whatever the quantity of knowledge, there isn’t a knowledge, or proxy knowledge, that’s ok or goal sufficient to adequately mannequin the underlying phenomenon. Such methods embody danger profiling that makes an attempt to foretell criminality, life course or the propensity to commit social safety fraud on the particular person stage or a selected location. These predictive methods have been debunked and decried as scientific malpractice.”
Focused surveillance feeds present biases
Laufer added that utilizing social knowledge to foretell if somebody will commit against the law inevitably targets people who belong to traditionally oppressed or marginalised teams which might be overrepresented within the underlying datasets, due to this fact exacerbating previous injustices.
“Makes an attempt to foretell fraud or criminality usually quantity to mechanically turning marginalised communities and people into suspects fairly than evidence-based decision-making,” he mentioned.
AI-powered instruments have beforehand been flagged by Amnesty as discriminatory, the place surveillance instruments had been getting used to trace and deport migrants within the US.
Petra Molnar, migration and human rights lawyer and director of the Refugee Regulation Lab at York College, beforehand advised Pc Weekly that AI-powered instruments can replicate present biases: “Algorithms are socially constructed, and our world is constructed on systemic racism and historic discrimination.”
Within the UK, Statewatch beforehand revealed that the UK Ministry of Justice was growing data-based profiling instruments to foretell crimes, together with a device aimed toward predicting potential murderers. Critics argued these methods worsen suggestions loops by growing surveillance of poor and racialised communities, with Amnesty Worldwide warning individually that predictive policing dangers “supercharging racism”.
In January 2026, the Dwelling Workplace introduced a £140m funding in PoliceAI – together with funding for 40 new stay facial recognition items – signalling a considerable scale-up of AI in policing set to roll out over the subsequent three years.
