The UK’s National Cyber Security Centre (NCSC) has issued a warning about the growing threat of “prompt injection” attacks targeting applications built using artificial intelligence (AI). While the warning is primarily directed at cybersecurity professionals working with large language models (LLMs) and other AI tools, it is important for all users of AI tools to understand prompt injection, as it is expected to become a significant category of security vulnerabilities in the future.
Prompt injection refers to an attack against LLMs, which are the language models that power chatbots like ChatGPT. In this type of attack, an attacker inserts a prompt in a way that bypasses any safeguards put in place by developers, thereby causing the AI to perform unauthorized actions. The potential damage caused by prompt injection attacks can range from generating harmful content to deleting important data or even conducting illicit financial transactions. The extent of the damage depends on the level of access the LLM has to external systems. While the risk of harm is relatively low for standalone chatbots, the NCSC warns that when developers integrate LLMs into existing applications, the potential for significant damage increases.
Attackers can gain control of LLMs through various means, including using jailbreak commands that deceive chatbots or other AI tools into responding positively to any prompt. For example, an LLM subjected to a suitable jailbreak prompt might provide detailed instructions on how to commit identity theft instead of refusing to do so. Additionally, there are indirect methods of prompt injection that create new categories of vulnerabilities.
In a proof of concept, security researcher Johann Rehberger successfully prompted ChatGPT to respond to a prompt embedded in a YouTube transcript. By using a plugin, Rehberger had ChatGPT summarize a video with a transcript containing specific instructions. Similarly, entrepreneur Cristiano Giardina created a website called “Bring Sydney Back” that concealed a prompt capable of revealing Bing chatbot’s secret alter ego. These examples highlight the security flaws present in LLMs, particularly those that integrate with applications and databases.
The NCSC provides an example of a bank that builds an LLM assistant to handle customer inquiries and instructions. In this scenario, an attacker could send a user a transaction request with a hidden prompt injection attack. When the user asks the chatbot about their spending for the month, the LLM analyzes transactions and encounters the malicious transaction, resulting in the attack reprogramming the LLM to transfer the user’s money to the attacker’s account.
Addressing prompt injection attacks is a complex problem with no easy solution. Existing AI-powered and filter-based approaches are insufficient, as they can only catch known attacks or a limited percentage of unknown attacks. The adversarial nature of security attacks means that determined attackers will continue to exploit vulnerabilities until they find a way to bypass existing defenses.
While there are suggestions for protecting LLM applications from prompt injection attacks, the reality is that LLMs and powerful AI chatbots are relatively new technologies, and their implications are not fully understood. The NCSC advises developers to treat LLMs as beta software—exciting to explore but not yet fully trustworthy.
In conclusion, prompt injection attacks pose a significant threat to applications built using AI. It is crucial for cybersecurity professionals and AI users to be aware of this vulnerability and take appropriate measures to mitigate the risks associated with prompt injection.