The UK’s cybersecurity agency, the National Cyber Security Centre (NCSC), has issued a warning about the potential risks of chatbots being manipulated by hackers. According to the NCSC, there is a growing cybersecurity threat of individuals using “prompt injection” attacks to manipulate chatbots and cause unintended consequences.
Chatbots, which rely on artificial intelligence, are designed to provide answers to user prompts and mimic human-like conversations. However, these language models can be vulnerable to manipulation if users create inputs or prompts that make them behave in unintended ways.
The NCSC has highlighted the risks associated with malicious prompt injection, particularly as chatbots are often used to pass data to third-party applications and services. If a user inputs a statement or question that the language model is not familiar with, or finds a combination of words to override the model’s original script or prompts, they can cause the chatbot to perform unintended actions.
These actions could range from generating offensive content to revealing confidential information in systems that accept unchecked input. The NCSC warns that such vulnerabilities could lead to attacks, scams, and data theft.
Recent incidents have demonstrated the potential risks. For example, a Stanford University student was able to uncover the hidden prompt of Microsoft’s Bing Chat by using a prompt injection. Similarly, a security researcher found that OpenAI’s ChatGPT could respond to new prompts through a third party, potentially leading to more indirect prompt injection vulnerabilities.
The NCSC emphasizes the importance of designing systems with security in mind to mitigate prompt injection attacks. Applying rules-based systems on top of machine learning models can help prevent damaging actions even when prompted to do so. By understanding the attack techniques that exploit inherent vulnerabilities in machine learning algorithms, cyber-attacks caused by artificial intelligence and machine learning can be mitigated.
As chatbots continue to be widely used in various industries, it is crucial for organizations to prioritize cybersecurity measures and stay vigilant against potential threats.