deepseek-ai / deepseek-ai/DeepSpec
Draft sampling uses a different distribution than speculative rejection
- Dominant language
- Python
- Stars
- 7.1k
- Forks
- 667
- PR merge metrics
- No merged PRs in 30d
Description
## Problem
`sample_tokens` samples draft tokens through the native-dtype logits path, while speculative rejection/correction uses the float32 distribution produced by `logits_to_probs`.
In speculative decoding, the proposal sample must be drawn from the same draft distribution used in the acceptance ratio and correction distribution. If the sample is drawn from `r` but the algorithm computes acceptance as if it came from `p`, the final marginal distribution is generally biased.
## Impact
With bfloat16 logits, the sampled draft distribution can differ from the float32 draft distribution used for rejection. That breaks the exact-distribution guarantee expected from speculative decoding.
## Fix
Fixed in PR #30 by casting logits to float32 before temperature scaling and softmax in `sample_tokens`, so sampling and rejection use the same draft distribution.
Contributor guide
No contributing guide indexed for this repository
Research direction
Review PR #30 alongside the sample_tokens and logits_to_probs entry points. Confirm that draft sampling and speculative rejection use the same float32 distribution, and verify the existing speculative-decoding checks for the exact-distribution guarantee.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 20/100