Britain’s National Cyber Security Centre (NCSC) has warned that organisations should be wary of integrating artificial intelligence-driven chatbots into their businesses due to potential security risks.

The UK cyber security agency has stated that despite understandable excitement around large language models (LLMs) like ChatGPT, Google Bard and Meta’s LLaMA, the global tech community still doesn’t yet fully understand LLM's capabilities, weaknesses, and (crucially) vulnerabilities.

“Whilst there are several LLM APIs already on the market, you could say our understanding of LLMs is still ‘in beta', albeit with a lot of ongoing global research helping to fill in the gaps,” the NCSC said in a blog post.

“However, organisations building services that use LLMs need to be careful, in the same way they would be if they were using a product or code library that was in beta

“They might not let that product be involved in making transactions on the customer's behalf, and hopefully wouldn't fully trust it yet. Similar caution should apply to LLMs.”

The NCSC said there were risks if such models were plugged into other elements of organization’s business processes. Academics and researchers have repeatedly found ways to subvert chatbots by feeding them rogue commands or fool them into circumventing their own built-in guardrails.

“One of the most widely reported weaknesses in the security of the current generation of LLMs is their vulnerability to 'prompt injection', which is when a user creates an input designed to make the model behave in an unintended way.

“This could mean causing it to generate offensive content, reveal confidential information, or trigger unintended consequences in a system that accepts unchecked input from the LLM. Hundreds of examples of prompt injection attacks have been published.

The NCSC said: “Prompt injection and data poisoning attacks can be extremely difficult to detect and mitigate.

“However, no model exists in isolation, so what we can do is design the whole system with security in mind. That is, by being aware of the risks associated with the ML [machine learning] component, we can design the system in such a way as to prevent exploitation of vulnerabilities leading to catastrophic failure.

“A simple example would be applying a rules-based system on top of the ML model to prevent it from taking damaging actions, even when prompted to do so.”

“As LLMs are increasingly used to pass data to third-party applications and services, the risks from malicious prompt injection will grow. At present, there are no failsafe security measures that will remove this risk. Consider your system architecture carefully and take care before introducing an LLM into a high-risk system.”