deepspeedai / deepspeedai/DeepSpeed

[BUG]GPU resources cannot be released properly when DeepSpeed is canceled. GPU A100 40G Driver Version: 535.54.03 CUDA Version: 12.2

Open
#4,003 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug training
Dominant language
Python
Stars
43.1k
Forks
5k
Avg merge
4d 15h
Merged PRs (30d)
112

Description

  1. GPU resources cannot be released properly when DeepSpeed is canceled.
  2. env Driver Version: 535.54.03 CUDA Version: 12.2
  3. log
    Jul 20 14:54:55 gpu02 kernel: INFO: rcu_sched detected stalls on CPUs/tasks: { 17} (detected by 40, t=60003 jiffies, g=86287, c=86286, q=237653)
    Jul 20 14:54:55 gpu02 kernel: Task dump for CPU 17:
    Jul 20 14:54:55 gpu02 kernel: AwsEventLoop 50 R running task 0 54782 1 0x0000008e
    Jul 20 14:54:55 gpu02 kernel: Call Trace:
    Jul 20 14:54:55 gpu02 kernel: [] unmap_sg+0x5f/0x70
    Jul 20 14:54:55 gpu02 kernel: [] nv_unmap_dma_map_scatterlist+0x8e/0xb0 [nvidia]
    Jul 20 14:54:55 gpu02 kernel: [] nv_dma_unmap_pages+0x115/0x120 [nvidia]
    Jul 20 14:54:55 gpu02 kernel: [] nv_dma_unmap_alloc+0x3d/0x60 [nvidia]
    Jul 20 14:54:55 gpu02 kernel: [] _nv039629rm+0xc5/0x1d0 [nvidia]
    Jul 20 14:54:55 gpu02 kernel: [] ? _nv031422rm+0x6f/0x90 [nvidia]
    Jul 20 14:54:55 gpu02 kernel: [] ? _nv034869rm+0x134/0x3a0 [nvidia]
    Jul 20 14:54:55 gpu02 kernel: [] ? _nv034834rm+0x6b/0x130 [nvidia]
    Jul 20 14:54:55 gpu02 kernel: [] ? _nv002511rm+0xd/0x20 [nvidia]
    Jul 20 14:54:55 gpu02 kernel: [] ? _nv004074rm+0x19/0xb0 [nvidia]
    Jul 20 14:54:55 gpu02 kernel: [] ? _nv016053rm+0x51c/0x620 [nvidia]
    Jul 20 14:54:55 gpu02 kernel: [] ? _nv043216rm+0xab/0xe0 [nvidia]
    Jul 20 14:54:55 gpu02 kernel: [] ? _nv044933rm+0xac/0x130 [nvidia]
    Jul 20 14:54:55 gpu02 kernel: [] ? _nv044932rm+0x2ef/0x690 [nvidia]
    Jul 20 14:54:55 gpu02 kernel: [] ? _nv044928rm+0xa7/0x1a0 [nvidia]
    Jul 20 14:54:55 gpu02 kernel: [] ? rm_cleanup_file_private+0x1db/0x200 [nvidia]
    Jul 20 14:54:55 gpu02 kernel: [] ? nvidia_close+0x13b/0x2f0 [nvidia]
    Jul 20 14:54:55 gpu02 kernel: [] ? nvidia_frontend_close+0x2f/0x50 [nvidia]
    Jul 20 14:54:55 gpu02 kernel: [] ? __fput+0xec/0x260
    Jul 20 14:54:55 gpu02 kernel: [] ? ____fput+0xe/0x10
    Jul 20 14:54:55 gpu02 kernel: [] ? task_work_run+0xbb/0xe0
    Jul 20 14:54:55 gpu02 kernel: [] ? do_exit+0x2d4/0xa50
    Jul 20 14:54:55 gpu02 kernel: [] ? timerqueue_del+0x24/0x70
    Jul 20 14:54:55 gpu02 kernel: [] ? __remove_hrtimer+0x3f/0xb0
    Jul 20 14:54:55 gpu02 kernel: [] ? do_group_exit+0x3f/0xa0
    Jul 20 14:54:55 gpu02 kernel: [] ? get_signal_to_deliver+0x1ce/0x5e0
    Jul 20 14:54:55 gpu02 kernel: [] ? do_signal+0x57/0x6f0
    Jul 20 14:54:55 gpu02 kernel: [] ? ep_poll+0x31e/0x360
    Jul 20 14:54:55 gpu02 kernel: [] ? wake_up_state+0x20/0x20
    Jul 20 14:54:55 gpu02 kernel: [] ? do_notify_resume+0x72/0xc0
    Jul 20 14:54:55 gpu02 kernel: [] ? int_signal+0x12/0x17

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No source files or tests are named. Start by reproducing cancellation of a DeepSpeed run on an A100 with Driver 535.54.03 and CUDA 12.2, then inspect the provided kernel log and process-cleanup path. Done means canceled work releases the GPU resources without the reported kernel stall.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, python
Domain
distributed-systems, machine-learning, operating-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
18/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.