The UK’s National Cyber Security Centre (NCSC) recently issued a warning about the increasing threat of “prompt injection” attacks on AI applications. While the warning is primarily aimed at cybersecurity professionals working with large language models (LLMs) and other AI tools, it is important for all users of AI tools to understand prompt injection, as it is expected to become a major category of security vulnerabilities in the future.
Prompt injection refers to an attack on LLMs, which are the language models that power chatbots like ChatGPT. In this type of attack, an attacker inserts a prompt in a way that bypasses any security measures put in place by developers, causing the AI to perform unauthorized actions. The potential damage caused by prompt injection attacks can range from outputting harmful content to deleting important information or conducting illicit financial transactions, depending on the level of access the LLM has to external systems. While the risk of harm is relatively low for standalone chatbots, the NCSC warns that when LLMs are integrated into existing applications, the potential for significant damage increases.
Attackers can gain control of LLMs through various methods, including using jailbreak commands that trick AI tools into responding affirmatively to any prompt. For example, an LLM that receives a suitable jailbreak prompt might provide detailed instructions on how to commit identity theft. Additionally, there are indirect methods of prompt injection that create new categories of vulnerabilities.
In a proof of concept earlier this year, security researcher Johann Rehberger was able to get ChatGPT to respond to a prompt embedded in a YouTube transcript. Another similar proof of concept involved entrepreneur Cristiano Giardina creating a website called “Bring Sydney Back” that could force the Bing chatbot sidebar to resurface its secret Sydney alter ego. These examples highlight the security flaws present in LLMs, particularly those that integrate with applications and databases.
The NCSC provides an example of a bank that builds an LLM assistant to handle customer inquiries and instructions. An attacker could send a user a transaction request with a hidden prompt injection attack on the LLM. When the user asks the chatbot about their spending for the month, the LLM analyzes transactions, encounters the malicious transaction, and is reprogrammed to send the user’s money to the attacker’s account.
Addressing prompt injection attacks is a challenging problem. Most AI-powered and filter-based approaches are ineffective in preventing such attacks. Building filters for known attacks or even catching 99% of unknown attacks is insufficient in the field of security. Adversarial attackers are highly motivated and will continue to exploit any vulnerabilities until they find a way to breach the system.
While there are ideas on how developers can protect their LLM applications from prompt injection attacks, the reality is that LLMs and powerful AI chatbots are still relatively new, and their behavior is not fully understood. The NCSC recommends treating LLMs as beta software, exciting to explore but not yet fully trustworthy.
In conclusion, prompt injection attacks pose a significant threat to AI applications built using LLMs. It is crucial for developers and users of AI tools to be aware of this vulnerability and take appropriate measures to mitigate the risks.