linkedin / linkedin/Liger-Kernel
Weighted Cross Entropy Loss
Open
- Dominant language
- Python
- Stars
- 6.6k
- Forks
- 603
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 47
Description
### 🚀 The feature, motivation and pitch
Allow for passing a weighting tensor to weight the CEL similar to C-RLFT where some tokens or inputs in the batch may have lower weighting to the overall loss. see https://arxiv.org/abs/2309.11235
### Alternatives
_No response_
### Additional context
_No response_
Contributor guide
Research direction
No files, tests, or entry points are identified in the issue. Start by locating the cross-entropy loss implementation and its tests, then review the linked C-RLFT paper; done means the loss accepts a weighting tensor for tokens or inputs and the behavior is covered by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100