antirez / antirez/ds4

ds4 parser vulnerability: user prompt can insert special tokens that are interpreted by the model

未关闭
#95 1 条评论 4 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
C
星标
22.3k
派生
2.1k
平均合并
1 天 3 小时
30 天内合并 PR
4

描述

*As previously agreed via email with @antirez, I am opening this issue to share the details of this vulnerability publicly.*

I'm reporting a vulnerability in the ds4 parser. The problem is the lack of separation between instructions (special tokens) and data (user prompt). Specifically, the [`tokenize_rendered_chat_vocab`](https://github.com/antirez/ds4/blob/011aa677a72486091ba5795791ef69e02347d2ae/ds4.c#L14766-L14785) function applies [`special_token_at`](https://github.com/antirez/ds4/blob/011aa677a72486091ba5795791ef69e02347d2ae/ds4.c#L14732-L14755) directly to the user prompt, without sanitizing it. This allows a user to inject special tokens, such as ``, `<|Assistant|>`, `|DSML|`, etc.

The attack scenario could be, for example, the attacker sending a prompt that creates a fake history for LLM. In this situation, the model can be fooled by the context (e.g., be tricked into making arbitrary tool calls, become misaligned, etc).

I isolated the parser logic in a standalone [PoC code](https://gist.github.com/pintorem/692b47edc8e043071fd6b8d51b09f913) (you don't need the necessary hardware to run it).

One possible fix is to ensure that `special_token_at` is only evaluated for strings generated by server logic, while forcing user input to pass exclusively through the standard BPE tokenizer.

If needed, I’d be more than happy to help by sending a pull request!

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。