deepseek-ai / deepseek-ai/DeepSpec
DSpark train/loss metric is rank-weighted instead of globally weighted
- Dominant language
- Python
- Stars
- 7.1k
- Forks
- 667
- PR merge metrics
- No merged PRs in 30d
Description
## Problem
The DSpark backward loss uses global denominators, but the logged `train/loss` metric was computed from locally normalized rank losses. Averaging those rank-local losses gives each rank equal weight, regardless of how many supervised tokens or objective terms each rank contributed.
## Impact
The logged loss can diverge from the actual global training objective. For example, a rank with one supervised token can affect the logged scalar as much as a rank with many supervised tokens.
## Fix
Fixed in PR #30 by logging the globally scaled `backward_loss.detach()` with `dp_mean`, so the reported loss matches the global objective used for gradients.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.