[BUG] Signal killed caused by Adam Offload
Open
Nobody has claimed this yet.
bug
- Dominant language
- Python
- Stars
- 624
- Forks
- 89
- PR merge metrics
- No merged PRs in 30d
Description
Is there an existing issue for this?
- I have searched the existing issues
Description of the Bug
When I try to run the finetune script of cpm-live-10b , adam offload optmizer caused the signal killed.And i tried to set the cpu level of adam to 0 which system wont use avx anymore.But it doesn't work.
Environment Information
- GCC version:9.4.0
- Torch version:1.13.0
- Linux system version:Ubuntu 20.04
- CUDA version:11.6+
- Torch's CUDA version (as per `torch.cuda.version()`):1.13 + 11.6
- CPU core: 48
To Reproduce
bash CPM-Bee/src/script/finetune_cpm_bee_10b.sh
Expected Behavior
It works normal
Screenshots
No response
Additional Information
No response
Confirmation
- I have reviewed and verified all the information provided in this report.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running CPM-Bee/src/script/finetune_cpm_bee_10b.sh in the reported Ubuntu, Torch, and CUDA environment and observe where Adam offload causes the process to be killed. Trace the optimizer behavior from that entry point and verify that the finetune completes without a signal-killed failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch, ubuntu
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100