Google warns criminals are building and selling illicit AI tools – and the market is growing

AI tools are being purpose built for criminals, new GTIG report finds
These tools side-step AI guardrails designed for safety
‘Just-in-time’ AI malware shows how criminals are evolving their techniques

Google’s Threat Intelligence Group has identified a worrying shift in AI trends, with AI no longer just being used to make criminals more productive, but also now being specially developed for active operations.

Its research found Large Language Models (LLMs) are being used in malware in particular, with ‘Just-in-Time’ AI like PROMPTFLUX – which is written in VBScript and engages with Gemini’s API to request ‘specific VBScript obfuscation and evasion techniques to facilitate “just-in-time” self-modification, likely to evade static signature-based detection’.

AI tools is maturing, and so is the criminal black market. Underground forums offer purpose-built AI tools that help lower the barrier for criminals to engage in illicit activities. This is bad news for everyone, since criminals no longer have to be particularly skilled to carry out complex cyberattacks, and they have a growing number of options.

Threat actors are using tactics reminiscent of social engineering to side-step AI safety features – pretending to be ‘cybersecurity researchers’ in order to convince Gemini to provide them with information that might otherwise be prohibited.

But who’s behind these incidents? Well, the research identifies, perhaps unsurprisingly, links to state-sponsored actors from Iran and China. These campaigns have a range of objectives, from data exfiltration to reconnaissance – similar to previously observed influence operations by the states, also using AI tools.

Since AI tools have become popularized, both criminals and security teams have been using the tools to boost productivity and assist in operations – and it’s not quite clear who has the upper hand.