[Serve] Disabling the overlap scheduler changes generation at temperature 0
Nobody has claimed this yet.
- Dominant language
- Mojo
- Stars
- 29.8k
- Forks
- 3.2k
- PR merge metrics
- No merged PRs in 30d
Description
Description
The same prompts produce materially different generations at temperature=0 depending on scheduler mode. On a 46-prompt classification benchmark (Qwen/Qwen3.5-4B, A10G, identical prompts and sampling params):
| Server flags | Accuracy | Avg latency |
|---|---|---|
| default (overlap scheduler ON) | 85% (39/46) | ~430 ms |
--no-enable-overlap-scheduler --force |
70% (32/46) | ~232 ms |
Reproducible back and forth: restoring the default scheduler restores 85%. The output distribution shifts noticeably (in our case the model stopped abstaining on low-signal inputs: 'unsure' recall dropped from 10/12 to 6/12). The large latency difference for identical workloads also looks suspicious.
At temperature 0 both modes should be near-deterministic and agree; a per-mode numerics or sampling divergence this large makes results measured in one mode invalid in the other — and since logprobs currently require the non-overlap mode, any logprob-based evaluation inherits the divergence.
Repro sketch: serve Qwen/Qwen3.5-4B --max-length 4096 --max-batch-size 1 with and without the flags above; send identical chat requests with temperature: 0 and compare outputs.
Assisted-by: AI
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the issue with Qwen/Qwen3.5-4B using the Serve command and the two scheduler flag sets, then compare temperature-0 outputs and latency on the stated prompts. Check the overlap scheduler and non-overlap path, including the logprob-required mode. Done means the scheduler modes no longer produce materially different generations for identical inputs, or the divergence is clearly characterized.
Written by the indexing model from the issue text.
Assessment
- Domain
- ai, machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100