Post ContentThe rise of AI has led to a surge in highly personalised scams. (Image: Unsplash)
Artificial intelligence (AI) helps unlock powerful new capabilities nearly every day, but its rapid progress continues to widen the scope for potential misuse. The latest addition to the list of AI-driven security risks is the de-anonymisation of data.
A new study has found that large language models (LLMs), the technology powering AI chatbots like ChatGPT, can be used by malicious actors to match anonymous users with their actual identities based on their posts on social media platforms.
The study, which has not been peer reviewed yet, was published last week by a group of researchers at ETH Zurich, Anthropic, and the Machine Learning Alignment and Theory Scholars programme. It shows that LLMs can be weaponised to carry out targeted attacks against users and violate their privacy, forcing a “fundamental reassessment of what can be considered private online.”
The findings of the study come at a time when online anonymity is under threat, not just from LLMs or AI agents but also due to the spread of age assurance technology, which has gained steam following global regulatory actions to ban children from using social media.
AI systems are also being increasingly used by governments for surveillance and military purposes. Recently, a high-stakes dispute erupted between Anthropic and the US government after the Claude maker doubled down on certain ‘red lines’ or restrictions that prevent the military or government use of its AI systems for domestic surveillance of US citizens.
Furthermore, the rise of AI has led to a surge in highly personalised scams as LLMs have significantly lowered the expertise requirement to carry out more sophisticated attacks against victims.
“Recent advances in LLM capabilities have made it clea that there is an urgent need to rethink various aspects of computer security in the wake of LLM-driven offensive cyber capabilities. Our work shows that the same is likely true for privacy as well,” the authors wrote.
Story continues below this ad
“We have demonstrated that LLMs enable deanonymisation of pseudonymous online accounts at scale, outperforming classical methods. In many cases, LLMs enable us to perform attacks that would not have previously been possible, due to the lack of structured data or features,” they added.
In order to test how effectively LLMs can be used to re-identify anonymised material online, the researchers developed an automated system of multiple AI agents using unspecified LLMs. These agents were designed to search the web and interact with information similar to the methods of most human investigators.
The AI agent-driven system treated posts or other texts as a set of clues that could be analysed for patterns related to someone’s identity such as writing quirks, stray biographical details, posting frequency and timing. The system then scanned potentially millions of user accounts on social media to match the mix of traits. It flagged probable matches and compared them to the clues in more detail.
Also Read | That ‘AI caricature using everything about me’ trend could expose you to digital fraud
The researchers said that they evaluated the multi-agent AI system using datasets built from publicly available posts, including content from Hacker News and LinkedIn, transcripts of Anthropic’s interviews with scientists on how they use AI, and Reddit accounts that were intentionally split into two anonymised halves for evaluation.
Story continues below this ad
In one case, the researchers gave the AI system a description of an anonymous account talking about school and walking their dog Biscuit through a “Dolores park”. The system then went through the process pipeline and accurately matched the anonymous account to the known identity.
However, the study noted that it is not a magic weapon against online anonymity as there might not be enough information for the AI system to draw conclusions. In many cases, the number of potential matches were also too large to narrow down.
