[Bug] Qwen-VL patch bypasses SGLang token-ID preservation and breaks multi-turn TITO
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 8.5k
- Forks
- 1.3k
- Avg merge
- 5h 36m
- Merged PRs (30d)
- 22
Description
Bug Description
Slime changes Qwen-VL from load_mm_data() to a direct legacy_load_mm_data() call:
This bypasses the code that preserves caller-provided token IDs.
In SGLang v0.5.15.post1, load_mm_data() first saves a list[int] prompt as input_ids, then
decodes the prompt for multimodal processing:
input_ids = prompt if isinstance(prompt, list) else None
prompt = self._tokenizer.decode(prompt)
If load_mm_data() needs the legacy loader, it forwards the saved IDs through the separate
input_ids=input_ids argument:
The Slime patch calls legacy_load_mm_data(prompt=input_text, ...) directly and does not pass its
separate input_ids argument. That argument therefore remains None:
The caller IDs are still present in prompt, but legacy_load_mm_data() immediately decodes that
list into text. It later returns the decoded/reconstructed text together with input_ids=None:
For raw images, process_and_combine_mm_data() then invokes the HF processor on that text and takes
the processor's newly generated ret["input_ids"]:
SGLang's protection against this retokenization requires base_output.input_ids is not None:
The caller must send pre-expansion rollout IDs: one image placeholder per image, together with the
exact token IDs returned by earlier generation turns. SGLang owns image-placeholder expansion for
its forward pass. The processor-expanded sequence can be retained separately for training.
Because the Slime patch leaves that field as None, the protection branch can never run. The raw
image path therefore always follows:
caller input_ids
-> decode to text
-> HF processor
-> newly tokenized input_ids
This means retokenization is executed for every affected request. The resulting IDs do not
necessarily differ on every request: canonical token sequences may round-trip unchanged. For a
non-canonical sequence, encode(decode(input_ids)) can differ from the caller's IDs, causing token
drift.
SGLang upstream added the preservation path specifically for this problem in
https://github.com/sgl-project/sglang/pull/26555
The fix is already present in the SGLang revision pinned by Slime, but the Slime patch bypasses it.
Steps to Reproduce
Run this CPU-only comparison in an SGLang v0.5.15.post1 source environment with the
Qwen/Qwen3.6-35B-A3B tokenizer. The first response ends with <|im_end|> and contains the valid
but non-canonical IDs [479, 3770] (["Ġtr", "uly"]) before it. Those two IDs decode to
" truly" and re-tokenize as the single ID [9149] (["Ġtruly"]). No SGLang source change or
model weights are required.
import asyncio
from difflib import SequenceMatcher
from transformers import AutoConfig, AutoProcessor
from sglang.srt.managers.schedule_batch import MultimodalInputFormat
from sglang.srt.multimodal.processors.base_processor import MultimodalSpecialTokens
from sglang.srt.multimodal.processors.qwen_vl import QwenVLImageProcessor
from sglang.srt.server_args import ServerArgs
async def main():
model_name = "Qwen/Qwen3.6-35B-A3B"
hf_processor = AutoProcessor.from_pretrained(model_name)
tokenizer = hf_processor.tokenizer
processor = QwenVLImageProcessor(
AutoConfig.from_pretrained(model_name),
ServerArgs(model_path=model_name),
hf_processor,
"default",
skip_mm_pool=True,
)
# Turn 1 prompt, including one image placeholder.
prompt_ids = tokenizer.encode(
"<|im_start|>user\nWhat is shown?"
"<|vision_start|><|image_pad|><|im_end|>\n"
"<|im_start|>assistant\n",
add_special_tokens=False,
)
# Turn 1 response exactly as generated by SGLang, including the assistant
# turn terminator. Each ID is annotated with its token and decoded text:
response_ids = [
479, # token "Ġtr", decoded fragment " tr"
3770, # token "uly", decoded fragment "uly"
248046, # token "<|im_end|>", decoded assistant-turn terminator
]
# Turn 2 starts after the complete response above.
next_prompt_ids = tokenizer.encode(
"\n<|im_start|>user\nExplain your answer.<|im_end|>\n"
"<|im_start|>assistant\n",
add_special_tokens=False,
)
accumulated_ids = prompt_ids + response_ids + next_prompt_ids
tokens = MultimodalSpecialTokens(
image_token="<|image_pad|>",
image_token_id=tokenizer.convert_tokens_to_ids("<|image_pad|>"),
)
tokens.parse_regex()
# Valid preprocessed input keeps this loader comparison CPU-only.
image_data = [{"format": MultimodalInputFormat.PROCESSOR_OUTPUT}]
try:
normal = await processor.load_mm_data(
prompt=accumulated_ids,
multimodal_tokens=tokens,
image_data=image_data,
)
legacy = await processor.legacy_load_mm_data(
prompt=accumulated_ids,
multimodal_tokens=tokens,
image_data=image_data,
)
_, normal_ids, _ = processor.process_and_combine_mm_data(normal, tokens)
_, legacy_ids, _ = processor.process_and_combine_mm_data(legacy, tokens)
finally:
processor.io_executor.shutdown()
processor.cpu_executor.shutdown()
normal_ids = normal_ids.tolist()
legacy_ids = legacy_ids.tolist()
print("response tokens:")
for index, token_id in enumerate(response_ids):
print(
f" {index}: id={token_id}, "
f"token={tokenizer.convert_ids_to_tokens(token_id)!r}, "
f"text={tokenizer.decode([token_id])!r}"
)
print("load_mm_data preserved caller IDs:", normal.input_ids == accumulated_ids)
print("legacy_load_mm_data input_ids:", legacy.input_ids)
print("normal final IDs preserved caller IDs:", normal_ids == accumulated_ids)
print(
"legacy decoded text matches caller:",
legacy.input_text == tokenizer.decode(accumulated_ids),
)
differences = []
for tag, caller_start, caller_end, legacy_start, legacy_end in SequenceMatcher(
a=accumulated_ids, b=legacy_ids, autojunk=False
).get_opcodes():
if tag == "equal":
continue
caller_span = accumulated_ids[caller_start:caller_end]
legacy_span = legacy_ids[legacy_start:legacy_end]
differences.append((tag, caller_span, legacy_span))
print(f"{tag}:")
print(
f" caller/load_mm_data[{caller_start}:{caller_end}]",
list(zip(caller_span, tokenizer.convert_ids_to_tokens(caller_span))),
)
print(
f" legacy retokenized[{legacy_start}:{legacy_end}]",
list(zip(legacy_span, tokenizer.convert_ids_to_tokens(legacy_span))),
)
assert normal.input_ids == accumulated_ids
assert legacy.input_ids is None
assert normal_ids == accumulated_ids
# Expected: the prior response's two IDs become one ID after retokenization.
assert differences == [("replace", [479, 3770], [9149])]
asyncio.run(main())
Expected output:
response tokens:
0: id=479, token='Ġtr', text=' tr'
1: id=3770, token='uly', text='uly'
2: id=248046, token='<|im_end|>', text='<|im_end|>'
load_mm_data preserved caller IDs: True
legacy_load_mm_data input_ids: None
normal final IDs preserved caller IDs: True
legacy decoded text matches caller: True
replace:
caller/load_mm_data[14:16] [(479, 'Ġtr'), (3770, 'uly')]
legacy retokenized[14:15] [(9149, 'Ġtruly')]
This matches the Slime patch exactly: it calls legacy_load_mm_data(prompt=input_text, ...) without
input_ids=input_text, so the original IDs are not retained for the anti-retokenization branch.
The original Qwen3-VL diagnosis captured the same pattern in three real multi-turn failures:
[350, 3140] ["ĠT", "CL"] -> [65231] ["ĠTCL"]
[1760, 529] ["Ġcount", "ert"] -> [60110] ["Ġcountert"]
[1841, 424] ["Ġsign", "age"] -> [79080] ["Ġsignage"]
Expected Behavior
For pre-tokenized image requests, the caller should send pre-expansion rollout IDs with one image
placeholder per image. SGLang should preserve every caller-provided non-image ID, including prior
response IDs, and expand the image placeholder for its forward pass.
Actual Behavior
The Slime patch loses the original-ID side channel. SGLang decodes the caller IDs and adopts the HF
processor's re-tokenized IDs, even though SGLANG_MM_AVOID_RETOKENIZE is enabled by default.
Environment
- slime commit:
4c193f1f37509cca70f0e88807a9305b70f63f4e - SGLang version: v0.5.15.post1
- OS: Linux
Logs
Additional Context
Suggested Fix
Remove the Qwen-VL load_mm_data() to legacy_load_mm_data() override from the active Slime SGLang
patches. load_mm_data() already falls back to legacy_load_mm_data() when needed while preserving
the original IDs through input_ids=input_ids.
The multi-turn VLM rollout should also keep pre-expansion rollout IDs separately from the
processor-expanded training IDs and send the rollout IDs to SGLang. The current recipe writes
processor output directly into sample.tokens and later sends that same sequence as input_ids:
Pre-submission Checklist
- I have read the CONTRIBUTING.md and understand the collaboration scope.
- I have read the documentation and my issue is not addressed there.
- I have searched for existing issues and this is not a duplicate.
- I have provided a minimal, reproducible example.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with docker/patch/latest/sglang.patch and examples/geo3k_vlm_multi_turn/rollout.py; compare the Qwen-VL override with SGLang’s load_mm_data() preservation path and trace how sample.tokens becomes rollout input_ids. Re-run the CPU-only reproducer with Qwen/Qwen3.6-35B-A3B and verify the override is removed, caller non-image IDs remain unchanged, and rollout and training sequences stay separate.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100