The UK’s cybersecurity agency, the National Cyber Security Centre (NCSC), has issued a warning about the potential risks associated with chatbots. According to the NCSC, hackers can manipulate chatbots through “prompt injection” attacks, which could lead to real-world consequences.
Chatbots, powered by artificial intelligence, are designed to respond to user prompts and mimic human-like conversations. However, these prompts can be manipulated to make the chatbot behave in unintended ways. This poses a cybersecurity risk, especially as chatbots are commonly used in online banking and shopping platforms.
Large language models (LLMs) like OpenAI’s ChatGPT and Google’s AI chatbot Bard are trained using data that generates human-like responses. As chatbots pass data to third-party applications and services, the NCSC warns that the risks of malicious prompt injection will increase.
If a user inputs a statement or question that the language model is unfamiliar with, or if they find a combination of words to override the model’s original script, the chatbot can perform unintended actions. This could result in the generation of offensive content or the disclosure of confidential information.
Instances of prompt injection attacks have already been identified. For example, a Stanford University student was able to uncover the entire prompt of Microsoft’s Bing Chat by requesting it to “ignore previous instructions.” Another security researcher found that ChatGPT could respond to new prompts through a third party that was not initially requested.
The NCSC emphasizes that prompt injection attacks can have real-world consequences if systems are not designed with security in mind. The vulnerability of chatbots and the ease with which prompts can be manipulated could lead to attacks, scams, and data theft.
To mitigate these risks, the NCSC suggests designing systems with security in mind and being aware of the vulnerabilities associated with machine learning components. Applying rules-based systems on top of machine learning models can prevent them from taking damaging actions even when prompted to do so.
By understanding the attack techniques that exploit inherent vulnerabilities in machine learning algorithms, cyber-attacks caused by artificial intelligence and machine learning can be mitigated.