linkedin / linkedin/Liger-Kernel

Weighted Cross Entropy Loss

Open
#404 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
6.6k
Forks
603
Avg merge
1d 20h
Merged PRs (30d)
47

Description

### 🚀 The feature, motivation and pitch

Allow for passing a weighting tensor to weight the CEL similar to C-RLFT where some tokens or inputs in the batch may have lower weighting to the overall loss. see https://arxiv.org/abs/2309.11235

### Alternatives

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are identified in the issue. Start by locating the cross-entropy loss implementation and its tests, then review the linked C-RLFT paper; done means the loss accepts a weighting tensor for tokens or inputs and the behavior is covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.