Please to fix the known bug in uvm_spin_loop()

NVIDIA Open GPU Kernel Modules Version

515.48.07

Does this happen with the proprietary driver (of the same version) as well?

Yes

Operating System and Version

CentOS Linux release 7.9.2009 (Core)

Kernel Release

3.10.0-1160.el7.x86_64

Hardware: GPU

Tesla P100-PCIE-16GB

Describe the bug

Execute pytorch program hangs on uvm_spin_loop() and can't kill it (even kill -9)

stack

[<ffffffffc33d41c2>] uvm_spin_loop+0xb2/0xe0 [nvidia_uvm]
[<ffffffffc3418ab3>] wait_for_entry_with_spin+0x63/0x170 [nvidia_uvm]
[<ffffffffc341912c>] uvm_tracker_wait_for_entry+0x4c/0x70 [nvidia_uvm]
[<ffffffffc3416f2c>] uvm_push_end_and_wait+0x4c/0x70 [nvidia_uvm]
[<ffffffffc33efa42>] channel_pool_add+0x512/0x8e0 [nvidia_uvm]
[<ffffffffc33eff14>] channel_manager_create_pools+0x104/0x1a0 [nvidia_uvm]
[<ffffffffc33f124c>] uvm_channel_manager_create+0xcc/0x360 [nvidia_uvm]
[<ffffffffc33e340b>] init_gpu+0x6cb/0xc40 [nvidia_uvm]
[<ffffffffc33e4dec>] add_gpu+0x7bc/0xdb0 [nvidia_uvm]
[<ffffffffc33e559a>] uvm_gpu_retain_by_uuid+0x1ba/0x230 [nvidia_uvm]
[<ffffffffc33e91ed>] uvm_va_space_register_gpu+0x3d/0x500 [nvidia_uvm]
[<ffffffffc33e68cc>] uvm_api_register_gpu+0x4c/0x70 [nvidia_uvm]
[<ffffffffc33d7da7>] uvm_ioctl+0xed7/0x1790 [nvidia_uvm]
[<ffffffffc33d869c>] uvm_unlocked_ioctl+0x3c/0x60 [nvidia_uvm]
[<ffffffffc33d87a4>] uvm_unlocked_ioctl_entry+0x64/0xd0 [nvidia_uvm]
[<ffffffffa38632e0>] do_vfs_ioctl+0x3a0/0x5b0
[<ffffffffa3863591>] SyS_ioctl+0xa1/0xc0
[<ffffffffa3d93f92>] system_call_fastpath+0x25/0x2a

Follow the code I saw a TODO in uvm_spin_loop()

// TODO: Bug 1710855: Also check fatal_signal_pending() here if the caller can handle it.

This is a known bug and maybe fix it in new version?

To Reproduce

        self.model.train()
        conf.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
        self.model = nn.DataParallel(self.model,device_ids=[0,1])
        self.model.to(conf.device)
        
         #############produce bug########################################
        self.head = nn.DataParallel(self.head,device_ids=[0,1])
        self.head.to(conf.device)
         #####################################################

Bug Incidence

Always

nvidia-bug-report.log.gz

nvidia-bug-report.log.gz

More Info

No response