Cannot access a tensor

Hello, I'm facing a pretty weird problem with the swin3dunet. I could run my training code for around 20 epochs on a v100 GPU, but eventually I will have the error below (I've set export CUDA_LAUNCH_BLOCKING=1 when running the code):

  File "/home/chenz0f/Swin-3D/decoder/backbone/swin3dunet.py", line 212, in forward
    sp, sp_down, coords_sp = layer(sp, coords_sp)
  File "/home/chenz0f/anaconda3/envs/v100/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/chenz0f/Swin-3D/decoder/backbone/modules/swin3d_layers.py", line 866, in forward
    sp_down, coords_sp = self.downsample(sp, coords_sp)
  File "/home/chenz0f/anaconda3/envs/v100/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/chenz0f/Swin-3D/decoder/backbone/modules/swin3d_layers.py", line 302, in forward
    feats = query_knn_feature(self.k, xyz, n_xyz, sp.F, offset, n_offset)
  File "/home/chenz0f/Swin-3D/decoder/backbone/modules/swin3d_layers.py", line 45, in query_knn_feature
    grouped_feat = src_feat[idx.view(-1).long(), :].view(m, K, c)
RuntimeError: CUDA error: device-side assert triggered
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

This error would occur together with several lines of CUDA assertion error message:

/opt/conda/conda-bld/pytorch_1678402411778/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [2318,0,0], thread: [0,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1678402411778/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [2318,0,0], thread: [1,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1678402411778/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [2318,0,0], thread: [2,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1678402411778/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [2318,0,0], thread: [3,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1678402411778/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [2318,0,0], thread: [4,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1678402411778/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [2318,0,0], thread: [5,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1678402411778/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [2318,0,0], thread: [6,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1678402411778/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [2318,0,0], thread: [7,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1678402411778/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [2318,0,0], thread: [8,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1678402411778/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [2318,0,0], thread: [9,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.

However, when I tried to check what was wrong, I was only able to check the shape of idx and src_feat. If I try to print their values, the CUDA assert error would also be triggered:

  File "/home/chenz0f/Swin-3D/decoder/backbone/modules/swin3d_layers.py", line 308, in forward
    feats = query_knn_feature(self.k, xyz, n_xyz, sp.F, offset, n_offset)
  File "/home/chenz0f/Swin-3D/decoder/backbone/modules/swin3d_layers.py", line 48, in query_knn_feature
    print("idx:", idx)
  File "/home/chenz0f/anaconda3/envs/v100/lib/python3.10/site-packages/torch/_tensor.py", line 426, in __repr__
    return torch._tensor_str._str(self, tensor_contents=tensor_contents)
  File "/home/chenz0f/anaconda3/envs/v100/lib/python3.10/site-packages/torch/_tensor_str.py", line 636, in _str
    return _str_intern(self, tensor_contents=tensor_contents)
  File "/home/chenz0f/anaconda3/envs/v100/lib/python3.10/site-packages/torch/_tensor_str.py", line 567, in _str_intern
    tensor_str = _tensor_str(self, indent)
  File "/home/chenz0f/anaconda3/envs/v100/lib/python3.10/site-packages/torch/_tensor_str.py", line 327, in _tensor_str
    formatter = _Formatter(get_summarized_data(self) if summarize else self)
  File "/home/chenz0f/anaconda3/envs/v100/lib/python3.10/site-packages/torch/_tensor_str.py", line 361, in get_summarized_data
    return torch.stack([get_summarized_data(x) for x in (start + end)])
  File "/home/chenz0f/anaconda3/envs/v100/lib/python3.10/site-packages/torch/_tensor_str.py", line 361, in <listcomp>
    return torch.stack([get_summarized_data(x) for x in (start + end)])
  File "/home/chenz0f/anaconda3/envs/v100/lib/python3.10/site-packages/torch/_tensor_str.py", line 353, in get_summarized_data
    return torch.cat(
RuntimeError: CUDA error: device-side assert triggered
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

I've also tried to use detach() and cpu(), but they all got the same error. May I ask if you've had similar problems before, or do you know what have might caused it?

Start in decoder/backbone/modules/swin3d_layers.py, especially query_knn_feature and the downsample call around line 302; inspect how idx, src_feat, xyz, and the offsets reach the failing indexed access. Reproduce with CUDA_LAUNCH_BLOCKING=1 and identify the condition producing an out-of-bounds index; done means a confirmed cause and a reproducible fix or validation.