Lightning-AI / Lightning-AI/pytorch-lightning
Gradient accumulation calcluation may be incorrect
Open
Nobody has claimed this yet.
bug
callback: gradient accumulation
ver: 2.4.x
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
Bug description
See https://unsloth.ai/blog/gradient for in depth explanation, and this PR for how huggingface fixed it.
I verified that I see worse performance with gradient accumulation than multiple devices, so I suspect this bug also applies to Lightning
What version are you seeing the problem on?
v2.4
How to reproduce the bug
No response
Error messages and logs
# Error messages and logs here please
Environment
Current environment
#- PyTorch Lightning Version (e.g., 2.4.0):
#- PyTorch Version (e.g., 2.4):
#- Python version (e.g., 3.12):
#- OS (e.g., Linux):
#- CUDA/cuDNN version:
#- GPU models and configuration:
#- How you installed Lightning(`conda`, `pip`, source):
More info
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Use the linked Unsloth explanation and Transformers PR as references, then reproduce the reported difference between gradient accumulation and multiple devices. Trace Lightning’s gradient-accumulation entry point; done means the discrepancy is covered by a regression test and the v2.4 behavior is corrected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100