deepspeedai / deepspeedai/DeepSpeed
Training accuracy heavily impacted when using activation checkpointing
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
I just use deepspeed without any additional params and let's get consistently 0.250 accuracy on 200'th batch. When I turn on the activation checkpointing, I get 0.025 accuracy with the same parameters at the same batch number.
My model is like an unet and has conv2d, leakyrelu and batchnorm layers and skip connections. Can this be a problem?
For skip connections I have modified a bit example from you and saved/consumed them when needed in a separate list
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No source file, test, or minimal reproduction is provided. First reproduce the accuracy gap with the stated UNet-like model and activation-checkpointing settings, then compare checkpointed and baseline runs; done means a confirmed cause with a regression test or documented limitation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100