根本缺陷使LLM极易受到攻击

研究人员发现LLM存在根本性缺陷（角色标签混淆），导致其无法完全抵御攻击，攻击者可诱导模型输出违禁信息。

• 攻击类型称为'思维链伪造'（chain-of-thought forgery），通过模仿模型内部思维链笔记来欺骗模型。
• 攻击示例：通过添加'用户身穿绿衬衫'的伪造思维链，使GPT-5输出制毒指南。
• 该攻击触发LLM的角色标签机制漏洞，模型无法区分用户指令与自身生成的思维链。
• 研究人员在OpenAI、Anthropic、阿里巴巴、DeepSeek的模型上均复现成功。
• 防御方法（红队测试与自动化模型）只能针对已知攻击，无法覆盖所有变体，因此无法彻底解决。

• There’s a real probability that this is going to be a problem that’s fundamentally unsolvable,” says Charles Ye, an independent researcher and coauthor of the ICML paper.
• writing instructions in a style that mimicked the text LLMs generate in their chain of thought—a kind of scratch pad that models use to write notes to themselves as they carry out tasks—would often trick the LLM into behaving as if it had come up with that instruction itself and acting on it.