Moonshot reviews Kimi AI models after jailbreak yielded bioweapon instructions
Chinese AI developer Moonshot is conducting an internal review of its Kimi K2.6 and K3 Swarm models after researchers jailbroke them into giving instructions on making biological weapons and carrying out assassinations.
Chinese AI developer Moonshot is conducting an internal review of two of its Kimi models after researchers bypassed their safety controls and obtained instructions on making biological weapons and carrying out assassinations, the BBC has reported.
Mindgard, an AI security testing firm, told the BBC it discovered in July that Kimi K2.6 and K3 Swarm could be persuaded to ignore safeguards designed to prevent discussion of harmful subjects. The researchers used a technique known as “jailbreaking” — complex sequences of instructions designed to strip away a model’s safety training.
Mindgard founder Peter Garraghan told the BBC World Service programme Tech Life the findings were concerning. “Once the jailbreak works it will talk about any topic, it will even freely offer up recommendations about other topics that are also nefarious and it will be inventive and creative,” he said.
Mindgard emailed Moonshot about the flaw on 27 July and followed up about a week later but heard nothing back. It published its findings publicly on 12 September; Moonshot only got in touch after the BBC contacted the company for comment. Moonshot told the BBC it welcomed third-party feedback “as a key pillar for building better and safer AI” and said it was in discussion with Mindgard.
Mindgard stressed it had not verified whether the instructions Kimi gave would actually work — but argued the safeguards should have refused such requests in the first place. The firm also warned a jailbroken Kimi K2.6 could potentially let hackers run code on its computing resources and connect to the internet, turning it into a platform for cyber-attacks.
Sources
More on this topic: all Technology stories