The rise of generative AI has been hailed as a game-changer for boosting productivity and cutting costs. However, concerns are growing about the potential weaponization of this technology by hackers to carry out devastating cyber attacks.
Generative AI, powered by large language models (LLMs), allows computers to generate complex content such as text, audio, video, and music from simple human inputs. It gained widespread attention with the launch of OpenAI’s ChatGPT, a consumer chatbot capable of answering user questions and commands.
Unfortunately, hackers have also taken notice. Following the release of ChatGPT, the number of posts on dark web forums discussing how to exploit the tool skyrocketed from 120 in January to 870 in February, marking a 625% increase, according to a report by NordVPN.
While most LLMs have filters to block harmful prompts, there is concern about prompt injections, where hackers find ways to bypass these filters and generate hate speech, propaganda, confidential information, or even malicious code. Numerous prompt injections have already been reported, with hackers and researchers testing whether systems can write computer malware and ransomware.
Mackenzie Jackson, developer advocate at cyber security group GitGuardian, explains that while the malware generated so far is relatively basic, generative AI is changing the landscape by enabling a new generation of “AI hackers” who know how to wield these off-the-shelf tools.
Eran Shimony, principal security researcher at CyberArk, has found ways to potentially use ChatGPT to automatically generate new families and types of malware. This creates “polymorphic” or mutating malware that can evade detection systems. Shimony highlights the ongoing cat-and-mouse game between hackers and chatbot owners, with hackers training their own models on malicious code to create more sophisticated and dangerous models.
Unfortunately, there are currently no foolproof ways to stop this, according to the National Cyber Security Centre. While research is ongoing, prompt injection may be an inherent issue with LLM technology. Additionally, the feedback loops used to develop open-source generative AI can be exploited by cyber criminals through data poisoning.
Furthermore, generative AI’s ability to automate tasks at scale poses additional cyber security risks. It could be used to write and deploy social engineering scams, targeting numerous victims in a more personalized manner than ever before.
Despite these concerns, Sandra Joyce, vice-president at Mandiant Intelligence, reassures that consequential hacks utilizing AI have not yet occurred. However, she notes the use of deepfakes in information or disinformation campaigns.
Defending against future AI-powered hacks will require increased vigilance and vetting. Some propose deploying AI in cyber defense more widely, using it to detect malware by identifying unusual behavior and patterns. Gartner’s recent report calls on cyber professionals to exploit GenAI opportunities for improved security and risk management.
Looking ahead, researchers are exploring the potential of fully automated cyber defense, where tools can detect new malware and instantly block its spread. This presents an opportunity to stay one step ahead of hackers.
However, Mackenzie Jackson warns that security teams will face a steep learning curve in the next five years. While we understand how hackers think, understanding the thought processes of these malicious programs remains a challenge.