deepseek-ai / deepseek-ai/DeepSpec

Eagle3 peaks at ~4.6 epochs then degrades; also 6% short of the released checkpoint (DSpark/DFlash reproduce)

Open
#84 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7.1k
Forks
667
PR merge metrics
No merged PRs in 30d

Description

Trained all three drafters from the unmodified configs in `config/` on the
documented data pipeline (Qwen3-4B target, full open-perfectblend split,
regen params from `scripts/data/README.md`, 8 GPUs, 10 epochs). Two things about
Eagle3.

### 1. It peaks around 4.6 epochs and degrades from there

Eval is 200 held-out prompts at `temperature=0` with a fixed seed, so it's
deterministic — re-running a checkpoint gives the same number.

```
epoch train loss accept_len
0.8 2.69 3.28
2.3 1.97 3.34
4.6 1.73 3.34 <- peak
7.6 1.59 3.24
10.0 1.50 3.18 <- final, -4.9%
```

Loss keeps going down the whole time. Four consecutive evals go down after the
peak and the per-position acceptance rates go down with them.

DSpark and DFlash off the same cache, same schedule: every per-1000-step
increment stays positive through the final step. Only Eagle3 turns over.

Also shows up on wall clock. Measured tokens/s against a plain autoregressive
baseline: our step-12000 checkpoint is ~1.5% off the released one's speedup, our
final checkpoint is ~8% off.

So the README line

> Each drafter is trained for 10 epochs to ensure full convergence.

holds for DSpark and DFlash here but not for Eagle3.

### 2. Even at its peak it's 6.2% short, while the other two land within 1%

```
ours (best ckpt) released ratio
DSpark 4.85 4.89 99.2%
DFlash 4.44 4.45 99.7%
Eagle3 3.34 3.56 93.8%
```

Best-to-released, not final-to-released, so this isn't double-counting the
overfitting above.

Checked and matching: architecture config (all keys, incl. `ttt_length`,
`step_loss_decay`, `draft_num_hidden_layers`, `target_layer_ids`), tensor
set/shapes/dtypes, bit-identical frozen `embed_tokens` and `lm_head`, regen
temperature/top-p/top-k/min-p/non-thinking, `min_loss_tokens`, config seed,
epoch count.

Two things ruled out by measuring rather than arguing:

- Not data-limited. Retrained Eagle3 on a ~170x smaller subset and compared
peaks: full data buys Eagle3 at most +1.10 accept_len, vs +2.30 for DSpark. It
benefits *less* from data, not more.
- Not missing regularisation. The released checkpoint's trained-weight norm is
~5% larger than our final's, and weight decay would make it smaller. Shipped
config is `weight_decay=0.0`.

Could still be something on our end, but DSpark and DFlash come out within 1% on
the same pipeline, so it looks specific to Eagle3.

### Questions

1. Are the released Eagle3 checkpoints the last step of a 10-epoch run, or picked
on a validation signal? README says "the direct output of the corresponding
training configuration", which reads like the former.
2. Do you see the turnover in 1 on your side? If so, the Table 1 Eagle3 row is
measured past its peak.
3. Anything in the Eagle3 training path not in `config/eagle3/eagle3_qwen3_4b.py`?
A `train_config.py` next to the released checkpoint, or just the hyperparams
used for it, would let us close 2 ourselves.
4. Worth documenting a shorter schedule for Eagle3, or adding val-based
checkpoint selection?

Let us know if you need more information. Looking forward to your reply.

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the Eagle3 run from config/eagle3/eagle3_qwen3_4b.py using the documented pipeline and scripts/data/README.md, then compare the training path and checkpoints with the released results. Verify the acceptance-length turnover and determine whether the released checkpoint used a validation-selected or shorter schedule. Done means the cause or missing configuration is documented, with a clear decision on checkpoint selection or Eagle3-specific training guidance.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.