deepspeedai / deepspeedai/DeepSpeed
[BUG]When fine-tuning an LLM, the following error occurs after training for some time: self.optimizer.param_groups[param_group_id]['params'] = [] IndexError: list index out of range
Open
@jomayeri is already working on this.
Since Dec 12, 2024.
bug
training
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
File "/share/home/xxx/miniconda3/envs/qwen2_vl_FT/lib/python3.10/site-packages/deepspeed/runtime/zero/stage3.py", line 2095, in step
[rank2]: self._optimizer_step(sub_group_id)
[rank2]: File "/share/home/xxx/miniconda3/envs/qwen2_vl_FT/lib/python3.10/site-packages/deepspeed/runtime/zero/stage3.py", line 980, in _optimizer_step
[rank2]: self.optimizer.param_groups[param_group_id]['params'] = []
[rank2]: IndexError: list index out of range
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.