deepseek-ai / deepseek-ai/DeepSpec
Warmup to cosine scheduler handoff can overshoot peak LR
- Dominant language
- Python
- Stars
- 7.1k
- Forks
- 667
- PR merge metrics
- No merged PRs in 30d
Description
## Problem
At the warmup/cosine transition, the scheduler wrapper can delegate to `CosineAnnealingLR.get_lr()` while the cosine scheduler is in a chainable state with `last_epoch == 0` and the optimizer LR already set to the warmup peak.
That state applies the recursive cosine update to the current peak LR instead of returning the epoch-0 peak value.
## Impact
The learning rate can exceed the configured peak for one step. With `eta_min = 0`, the transition factor is:
```text
2 / (1 + cos(pi / T_max))
```
For example, `T_max = 10` overshoots by about 2.51%.
## Fix
Fixed in PR #30 by special-casing the handoff to return the configured peak/base LRs directly, then letting the following scheduler step advance cosine normally.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.