The dark web is home to a growing array of artificial-intelligence chatbots similar to ChatGPT, but designed to help hackers. Businesses are on high alert for a glut of AI-generated email fraud and deepfakes.
5 min
A new crop of nefarious chatbots with names like “BadGPT” and “FraudGPT” are springing up on the darkest corners of the web, as cybercriminals look to tap the same artificial intelligence behind OpenAI’s ChatGPT.
Just as some office workers use ChatGPT to write better emails, hackers are using manipulated versions of AI chatbots to turbocharge their phishing emails. They can use chatbots—some also freely-available on the open internet—to create fake websites, write malware and tailor messages to better impersonate executives and other trusted entities.
Earlier this year, a Hong Kong multinational company employee handed over $25.5 million to an attacker who posed as the company’s chief financial officer on an AI-generated deepfake conference call, the South China Morning Post reported, citing Hong Kong police. Chief information officers and cybersecurity leaders, already accustomed to a growing spate of cyberattacks , say they are on high alert for an uptick in more sophisticated phishing emails and deepfakes.
Vish Narendra, CIO of Graphic Packaging International, said the Atlanta-based paper packing company has seen an increase in what are likely AI-generated email attacks called spear-phishing , where cyber attackers use information about a person to make an email seem more legitimate. Public companies in the spotlight are even more susceptible to contextualised spear-phishing, he said.
Researchers at Indiana University recently combed through over 200 large-language model hacking services being sold and populated on the dark web. The first service appeared in early 2023—a few months after the public release of OpenAI’s ChatGPT in November 2022.
Most dark web hacking tools use versions of open-source AI models like Meta ’s Llama 2, or “jailbroken” models from vendors like OpenAI and Anthropic to power their services, the researchers said. Jailbroken models have been hijacked by techniques like “ prompt injection ” to bypass their built-in safety controls.
Jason Clinton, chief information security officer of Anthropic, said the AI company eliminates jailbreak attacks as they find them, and has a team monitoring the outputs of its AI systems. Most model-makers also deploy two separate models to secure their primary AI model, making the likelihood that all three will fail the same way “a vanishingly small probability.”
Meta spokesperson Kevin McAlister said that openly releasing models shares the benefits of AI widely, and allows researchers to identify and help fix vulnerabilities in all AI models, “so companies can make models more secure.”
An OpenAI spokesperson said the company doesn’t want its tools to be used for malicious purposes, and that it is “always working on how we can make our systems more robust against this type of abuse.”
Malware and phishing emails written by generative AI are especially tricky to spot because they are crafted to evade detection. Attackers can teach a model to write stealthy malware by training it with detection techniques gleaned from cybersecurity defence software, said Avivah Litan, a generative AI and cybersecurity analyst at Gartner.
Phishing emails grew by 1,265% in the 12-month period starting when ChatGPT was publicly released, with an average of 31,000 phishing attacks sent every day, according to an October 2023 report by cybersecurity vendor SlashNext.
“The hacking community has been ahead of us,” said Brian Miller, CISO of New York-based not-for-profit health insurer Healthfirst, which has seen an increase in attacks impersonating its invoice vendors over the past two years.
While it is nearly impossible to prove whether certain malware programs or emails were created with AI, tools developed with AI can scan for text likely created with the technology. Abnormal Security , an email security vendor, said it had used AI to help identify thousands of likely AI-created malicious emails over the past year, and that it had blocked a twofold increase in targeted, personalised email attacks.
When Good Models Go Bad
Part of the challenge in stopping AI-enabled cybercrime is some AI models are freely shared on the open web. To access them, there is no need for dark corners of the internet or exchanging cryptocurrency.
Such models are considered “uncensored” because they lack the enterprise guardrails that businesses look for when buying AI systems, said Dane Sherrets, an ethical hacker and senior solutions architect at bug bounty company HackerOne.
In some cases, uncensored versions of models are created by security and AI researchers who strip out their built-in safeguards. In other cases, models with safeguards intact will write scam messages if humans avoid obvious triggers like “phishing”—a situation Andy Sharma, CIO and CISO of Redwood Software, said he discovered when creating a spear-phishing test for his employees.
The most useful model for generating scam emails is likely a version of Mixtral, from French AI startup Mistral AI, that has been altered to remove its safeguards, Sherrets said. Due to the advanced design of the original Mixtral, the uncensored version likely performs better than most dark web AI tools, he added. Mistral did not reply to a request for comment.
Sherrets recently demonstrated the process of using an uncensored AI model to generate a phishing campaign. First, he searched for “uncensored” models on Hugging Face, a startup that hosts a popular repository of open-source models—showing how easily many can be found.
He then used a virtual computing service that cost less than $1 per hour to mimic a graphics processing unit, or GPU, which is an advanced chip that can power AI. A bad actor needs either a GPU or a cloud-based service to use an AI model, Sherrets said, adding that he learned most of how to do this on X and YouTube.
With his uncensored model and virtual GPU service running, Sherrets asked the bot: “Write a phishing email targeting a business that impersonates a CEO and includes publicly-available company data,” and “Write an email targeting the procurement department of a company requesting an urgent invoice payment.”
The bot sent back phishing emails that were well-written, but didn’t include all of the personalisation asked for. That’s where prompt engineering , or the human’s ability to better extract information from chatbots, comes in, Sherrets said.
Dark Web AI Tools Can Already Do Harm
For hackers, a benefit of dark web tools like BadGPT—which researchers said uses OpenAI’s GPT model—is that they are likely trained on data from those underground marketplaces. That means they probably include useful information like leaks, ransomware victims and extortion lists, said Joseph Thacker, an ethical hacker and principal AI engineer at cybersecurity software firm AppOmni.
While some underground AI tools have been shuttered, new services have already taken their place, said Indiana University Assistant Computer Science Professor Xiaojing Liao, a co-author of the study. The AI hacking services, which often take payment via cryptocurrency, are priced anywhere from $5 to $199 a month.
New tools are expected to improve just as the AI models powering them do. In a matter of years, AI-generated text, video and voice deepfakes will be virtually indistinguishable from their human counterparts, said Evan Reiser , CEO and co-founder of Abnormal Security.
While researching the hacking tools, Indiana University Associate Dean for Research XiaoFeng Wang, a co-author of the study, said he was surprised by the ability of dark web services to generate effective malware. Given just the code of a security vulnerability, the tools can easily write a program to exploit it.
Though AI hacking tools often fail, in some cases, they work. “That demonstrates, in my opinion, that today’s large language models have the capability to do harm,” Wang said.
Copyright 2020, Dow Jones & Company, Inc. All Rights Reserved Worldwide. LEARN MORE
What a quarter-million dollars gets you in the western capital.
Alexandre de Betak and his wife are focusing on their most personal project yet.
Multinationals like Starbucks and Marriott are taking a hard look at their Chinese operations—and tempering their outlooks.
4 min
For years, global companies showcased their Chinese operations as a source of robust growth. A burgeoning middle class, a stream of people moving to cities, and the creation of new services to cater to them—along with the promise of the further opening of the world’s second-largest economy—drew companies eager to tap into the action.
Then Covid hit, isolating China from much of the world. Chinese leader Xi Jinping tightened control of the economy, and U.S.-China relations hit a nadir. After decades of rapid growth, China’s economy is stuck in a rut, with increasing concerns about what will drive the next phase of its growth.
Though Chinese officials have acknowledged the sputtering economy, they have been reluctant to take more than incremental steps to reverse the trend. Making matters worse, government crackdowns on internet companies and measures to burst the country’s property bubble left households and businesses scarred.
Now, multinational companies are taking a hard look at their Chinese operations and tempering their outlooks. Marriott International narrowed its global revenue per available room growth rate to 3% to 4%, citing continued weakness in China and expectations that demand could weaken further in the third quarter. Paris-based Kering , home to brands Gucci and Saint Laurent, posted a 22% decline in sales in the Asia-Pacific region, excluding Japan, in the first half amid weaker demand in Greater China, which includes Hong Kong and Macau.
Pricing pressure and deflation were common themes in quarterly results. Starbucks , which helped build a coffee culture in China over the past 25 years, described it as one of its most notable international challenges as it posted a 14% decline in sales from that business. As Chinese consumers reconsidered whether to spend money on Starbucks lattes, competitors such as Luckin Coffee increased pressure on the Seattle company. Starbucks executives said in their quarterly earnings call that “unprecedented store expansion” by rivals and a price war hurt profits and caused “significant disruptions” to the operating environment.
Executive anxiety extends beyond consumer companies. Elevator maker Otis Worldwide saw new-equipment orders in China fall by double digits in the second quarter, forcing it to cut its outlook for growth out of Asia. CEO Judy Marks told analysts on a quarterly earnings call that prices in China were down roughly 10% year over year, and she doesn’t see the pricing pressure abating. The company is turning to productivity improvements and cost cutting to blunt the hit.
Add in the uncertainty created by deteriorating U.S.-China relations, and many investors are steering clear. The iShares MSCI China exchange-traded fund has lost half its value since March 2021. Recovery attempts have been short-lived. undefined undefined And now some of those concerns are creeping into the U.S. market. “A decade ago China exposure [for a global company] was a way to add revenue growth to our portfolio,” says Margaret Vitrano, co-manager of large-cap growth strategies at ClearBridge Investments in New York. Today, she notes, “we now want to manage the risk of the China exposure.”
Vitrano expects improvement in 2025, but cautions it will be slow. Uncertainty over who will win the U.S. presidential election and the prospect of higher tariffs pose additional risks for global companies.
For now, China is inching along at roughly 5% economic growth—down from a peak of 14% in 2007 and an average of about 8% in the 10 years before the pandemic. Chinese consumers hit by job losses and continued declines in property values are rethinking spending habits. Businesses worried about policy uncertainty are reluctant to invest and hire.
The trouble goes beyond frugal consumers. Xi is changing the economy’s growth model, relying less on the infrastructure and real estate market that fueled earlier growth. That means investing aggressively in manufacturing and exports as China looks to become more self-reliant and guard against geopolitical tensions.
The shift is hurting western multinationals, with deflationary forces amid burgeoning production capacity. “We have seen the investment community mark down expectations for these companies because they will have to change tack with lower-cost products and services,” says Joseph Quinlan, head of market strategy for the chief investment office at Merrill and Bank of America Private Bank.
Another challenge for multinationals outside of China is stiffened competition as Chinese companies innovate and expand—often with the backing of the government. Local rivals are upping the ante across sectors by building on their knowledge of local consumer preferences and the ability to produce higher-quality products.
Some global multinationals are having a hard time keeping up with homegrown innovation. Auto makers including General Motors have seen sales tumble and struggled to turn profitable as Chinese car shoppers increasingly opt for electric vehicles from BYD or NIO that are similar in price to internal-combustion-engine cars from foreign auto makers.
“China’s electric-vehicle makers have by leaps and bounds surpassed the capabilities of foreign brands who have a tie to the profit pool of internal combustible engines that they don’t want to disrupt,” says Christine Phillpotts, a fund manager for Ariel Investments’ emerging markets strategies.
Chinese companies are often faster than global rivals to market with new products or tweaks. “The cycle can be half of what it is for a global multinational with subsidiaries that need to check with headquarters, do an analysis, and then refresh,” Phillpotts says.
For many companies and investors, next year remains a question mark. Ashland CEO Guillermo Novo said in an August call with analysts that the chemical company was seeing a “big change” in China, with activity slowing and competition on pricing becoming more aggressive. The company, he said, was still trying to grasp the repercussions as it has created uncertainty in its 2025 outlook.
Few companies are giving up. Executives at big global consumer and retail companies show no signs of reducing investment, with most still describing China as a long-term growth market, says Dana Telsey, CEO of Telsey Advisory Group.
Starbucks executives described the long-term opportunity as “significant,” with higher growth and margin opportunities in the future as China’s population continues to move from rural to suburban areas. But they also noted that their approach is evolving and they are in the early stages of exploring strategic partnerships.
Walmart sold its stake in August in Chinese e-commerce giant JD.com for $3.6 billion after an eight-year noncompete agreement expired. Analysts expect it to pump the money into its own Sam’s Club and Walmart China operation, which have benefited from the trend toward trading down in China.
“The story isn’t over for the global companies,” Phillpotts says. “It just means the effort and investment will be greater to compete.”
Corrections & Amplifications
Joseph Quinlan is head of market strategy for the chief investment office at Merrill and Bank of America Private Bank. An earlier version of this article incorrectly used his old title.
