deepseek-ai / deepseek-ai/DeepSpec

Warmup to cosine scheduler handoff can overshoot peak LR

Open
#32 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7.1k
Forks
667
PR merge metrics
No merged PRs in 30d

Description

## Problem

At the warmup/cosine transition, the scheduler wrapper can delegate to `CosineAnnealingLR.get_lr()` while the cosine scheduler is in a chainable state with `last_epoch == 0` and the optimizer LR already set to the warmup peak.

That state applies the recursive cosine update to the current peak LR instead of returning the epoch-0 peak value.

## Impact

The learning rate can exceed the configured peak for one step. With `eta_min = 0`, the transition factor is:

```text
2 / (1 + cos(pi / T_max))
```

For example, `T_max = 10` overshoots by about 2.51%.

## Fix

Fixed in PR #30 by special-casing the handoff to return the configured peak/base LRs directly, then letting the following scheduler step advance cosine normally.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.