AI Wrote Better Phishing Emails Than Humans in a Recent Test

Natural language processing continues to find its way into unexpected corners. This time, it’s phishing emails. In a small study, researchers found that they could use the deep learning language model GPT-3, along with other AI-as-a-service platforms, to significantly lower the barrier to entry for crafting spearphishing campaigns at a massive scale. 

Researchers have long debated whether it would be worth the effort for scammers to train machine learning algorithms that could then generate compelling phishing messages. Mass phishing messages are simple and formulaic, after all, and are already highly effective. Highly targeted and tailored “spearphishing” messages are more labor intensive to compose, though. That’s where NLP may come in surprisingly handy.

At the Black Hat and Defcon security conferences in Las Vegas this week, a team from Singapore’s Government Technology Agency presented a recent experiment in which they sent targeted phishing emails they crafted themselves and others generated by an AI-as-a-service platform to 200 of their colleagues. Both messages contained links that were not actually malicious but simply reported back clickthrough rates to the researchers. They were surprised to find that more people clicked the links in the AI-generated messages than the human-written ones—by a significant margin.

“Researchers have pointed out that AI requires some level of expertise. It takes millions of dollars to train a really good model,” says Eugene Lim, a Government Technology Agency cybersecurity specialist. “But once you put it on AI-as-a-service it costs a couple of cents and it’s really easy to use—just text in, text out. You don’t even have to run code, you just give it a prompt and it will give you output. So that lowers the barrier of entry to a much bigger audience and increases the potential targets for spearphishing. Suddenly every single email on a mass scale can be personalized for each recipient.”

The researchers used OpenAI’s GPT-3 platform in conjunction with other AI-as-a-service products focused on personality analysis to generate phishing emails tailored to their colleagues’ backgrounds and traits. Machine learning focused on personality analysis aims to be predict a person’s proclivities and mentality based on behavioral inputs. By running the outputs through multiple services, the researchers were able to develop a pipeline that groomed and refined the emails before sending them out. They say that the results sounded “weirdly human” and that the platforms automatically supplied surprising specifics, like mentioning a Singaporean law when instructed to generate content for people living in Singapore.

While they were impressed by the quality of the synthetic messages and how many clicks they garnered from colleagues versus the human-composed ones, the researchers note that the experiment was just a first step. The sample size was relatively small and the target pool was fairly homogenous in terms of employment and geographic region. Plus, both the human-generated messages and those generated by the AI-as-a-service pipeline were created by office insiders rather than outside attackers trying to strike the right tone from afar.

“There are lots of variables to account for,” says Tan Kee Hock, a Government Technology Agency cybersecurity specialist.

Still, the findings spurred the researchers to think more deeply about how AI-as-a-service may play a role in phishing and spearphishing campaigns moving forward. OpenAI itself, for example, has long feared the potential for misuse of its own service or other similar ones. The researchers note that it and other scrupulous AI-as-a-service providers have clear codes of conduct, attempt to audit their platforms for potentially malicious activity, or even try to verify user identities to some degree. 

“Misuse of language models is an industry-wide issue that we take very seriously as part of our commitment to the safe and responsible deployment of AI,” OpenAI told WIRED in a statement. “We grant access to GPT-3 through our API, and we review every production use of GPT-3 before it goes live. We impose technical measures, such as rate limits, to reduce the likelihood and impact of malicious use by API users. Our active monitoring systems and audits are designed to surface potential evidence of misuse at the earliest possible stage, and we are continually working to improve the accuracy and effectiveness of our safety tools.”