deepseek-ai / deepseek-ai/DeepSpec

Draft sampling uses a different distribution than speculative rejection

Open
#31 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7.1k
Forks
667
PR merge metrics
No merged PRs in 30d

Description

## Problem

`sample_tokens` samples draft tokens through the native-dtype logits path, while speculative rejection/correction uses the float32 distribution produced by `logits_to_probs`.

In speculative decoding, the proposal sample must be drawn from the same draft distribution used in the acceptance ratio and correction distribution. If the sample is drawn from `r` but the algorithm computes acceptance as if it came from `p`, the final marginal distribution is generally biased.

## Impact

With bfloat16 logits, the sampled draft distribution can differ from the float32 draft distribution used for rejection. That breaks the exact-distribution guarantee expected from speculative decoding.

## Fix

Fixed in PR #30 by casting logits to float32 before temperature scaling and softmax in `sample_tokens`, so sampling and rejection use the same draft distribution.

Contributor guide

No contributing guide indexed for this repository

Research direction

Review PR #30 alongside the sample_tokens and logits_to_probs entry points. Confirm that draft sampling and speculative rejection use the same float32 distribution, and verify the existing speculative-decoding checks for the exact-distribution guarantee.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.