The UK’s National Cyber Security Centre (NCSC) recently issued a warning about the growing threat of “prompt injection” attacks against AI applications. Prompt injection attacks target large language models (LLMs) and other AI tools, and they have the potential to cause significant damage. These attacks involve inserting a prompt that bypasses the developers’ safeguards, allowing the AI to perform unauthorized actions such as outputting harmful content or conducting illicit transactions.
While prompt injection attacks are primarily aimed at LLMs used in chatbots like ChatGPT, they pose a risk to any AI tool. Attackers can take control of LLMs by using jailbreak commands that trick the AI into responding affirmatively to any prompt. Additionally, there are indirect methods of prompt injection that create new vulnerabilities.
Security researcher Johann Rehberger demonstrated a proof of concept by getting ChatGPT to respond to a prompt embedded in a YouTube transcript. Another proof of concept involved a website called “Bring Sydney Back,” which could force the Bing chatbot sidebar to resurface its secret alter ego. These examples highlight the security flaws present in LLMs, especially when integrated with applications and databases.
The NCSC warns that prompt injection attacks could have serious consequences, such as an attacker reprogramming an LLM assistant at a bank to send a user’s money to the attacker’s account. Similarly, if an AI assistant can read emails, it can be tricked into responding to prompts embedded in malicious messages.
Unfortunately, prompt injection is a challenging problem to solve. Most AI-powered and filter-based approaches are ineffective against these attacks. The NCSC recommends treating LLMs as beta software and not fully trusting them until more robust security measures are developed.
While developers explore ways to protect their LLM applications from prompt injection attacks, it is clear that the landscape of powerful AI chatbots and LLMs is still evolving. The NCSC acknowledges that there is much to learn and understand about these technologies, and caution should be exercised when implementing them.