huggingface / huggingface/picotron

Mistake in Ring Attention Backward

Open
#37 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.3k
Forks
203
PR merge metrics
No merged PRs in 30d

Description

There is a mistake in backward pass in computing dLoss/d(softmax).
dL / d(softmax) = P * (dP - row_sum(dP * P)) but the code mistakenly did dL / d(softmax) = P * (dP - row_sum(dO * O)).

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the ring-attention backward-pass implementation and read how dLoss/d(softmax) is computed. Compare the implementation with the derivative stated in the issue, then verify that the backward calculation uses the stated row sum and that the existing project checks pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.