The UK’s National Cyber Security Centre (NCSC) has issued a warning about the increasing threat of “prompt injection” attacks on AI applications. Prompt injection attacks target large language models (LLMs) like ChatGPT, which power chatbots. In these attacks, an attacker inserts a prompt to bypass security measures and manipulate the AI into performing unauthorized actions. The potential damage ranges from outputting harmful content to conducting illicit financial transactions. While the risk is low for standalone chatbots, the NCSC emphasizes that prompt injection attacks become more significant when LLMs are integrated into existing applications.
Attackers can gain control over LLMs by using jailbreak commands that deceive the AI into responding affirmatively to any prompt. For example, an LLM hit with a suitable jailbreak prompt may provide detailed instructions on committing identity theft. Additionally, there are various methods of “indirect prompt injection” that pose new challenges.
Security researcher Johann Rehberger demonstrated a proof of concept by getting ChatGPT to respond to a prompt embedded in a YouTube transcript. Another proof of concept by entrepreneur Cristiano Giardina involved a hidden prompt on a website that forced the Bing chatbot sidebar to reveal its secret alter ego. These examples highlight the security flaws in LLMs, particularly those integrated with applications and databases.
The NCSC provides an example of a bank using an LLM assistant to handle customer inquiries and instructions. An attacker could hide a prompt injection attack within a transaction request, causing the LLM to redirect funds to the attacker’s account. Similarly, if an AI assistant like Marvin can read emails, attackers could send prompts instructing it to forward sensitive information.
Unfortunately, prompt injection is a challenging problem to solve. Traditional AI-powered and filter-based approaches are insufficient because attackers constantly find ways to bypass them. While there are suggestions for protecting LLM applications, the novelty of LLMs and powerful AI chatbots means that the future remains uncertain. The NCSC recommends treating LLMs as beta software, exciting to explore but not yet fully trustworthy.