The UK’s National Cyber Security Centre (NCSC) has issued a warning about the increasing threat of “prompt injection” attacks on AI applications. While the warning is primarily aimed at cybersecurity professionals working with large language models (LLMs) and other AI tools, it is important for all users of AI technology to be aware of this potential security vulnerability.
Prompt injection attacks involve attackers inserting prompts into LLMs in a way that bypasses any safeguards put in place by developers, allowing the AI to perform unauthorized actions. The extent of the damage caused by these attacks depends on the level of access the LLM has to external systems. While the risk is relatively low for standalone chatbots, the NCSC warns that when LLMs are integrated into existing applications, the potential for significant harm increases.
Attackers can gain control of LLMs by using jailbreak commands that trick the AI into responding positively to any prompt. This can lead to instructions being given for harmful activities such as identity theft or illicit financial transactions. There are also indirect methods of prompt injection that create additional security vulnerabilities.
Security researchers have demonstrated proof of concept attacks, including one where a YouTube transcript was used to prompt an AI chatbot to respond affirmatively to an attack. Another example involved a website that forced a chatbot sidebar to reveal its hidden alter ego. These examples highlight the flaws in LLMs, particularly when they interact with applications and databases.
The NCSC provides an example of a bank that builds an LLM assistant for handling customer queries and instructions. In this scenario, an attacker could send a transaction request with a hidden prompt injection attack, causing the LLM to transfer money to the attacker’s account when a user asks about their spending habits.
Addressing prompt injection attacks is a challenging problem. Existing AI-powered and filter-based approaches are not sufficient, as attackers constantly find new ways to bypass them. While there are suggestions for protecting LLM applications from these attacks, the reality is that LLMs and powerful AI chatbots are still relatively new, and their behavior is not fully understood.
The NCSC advises developers to treat LLMs as beta software, exciting to explore but not yet fully trustworthy. As the field of AI continues to evolve, it is crucial for cybersecurity professionals and users alike to stay vigilant and adapt to emerging threats.