RuntimeError: nvrtc: error: invalid value for --gpu-architecture (-arch)

the environment is :
------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------
sys.platform linux
Python 3.8.19 (default, Mar 20 2024, 19:58:24) [GCC 11.2.0]
numpy 1.24.3
detectron2 0.6 @/media/hky/d2bed7b9-228d-4a5b-ae76-f3f34ce12c7b/hky/lsj/code/MASKDINO/detectron2-main/detectron2
Compiler GCC 7.5
CUDA compiler CUDA 11.3
detectron2 arch flags /media/hky/d2bed7b9-228d-4a5b-ae76-f3f34ce12c7b/hky/lsj/code/MASKDINO/detectron2-main/detectron2/_C.cpython-38-x86_64-linux-gnu.so; cannot find cuobjdump
DETECTRON2_ENV_MODULE
PyTorch 1.10.0 @/media/hky/d2bed7b9-228d-4a5b-ae76-f3f34ce12c7b/hky/anaconda3/envs/maskdino/lib/python3.8/site-packages/torch
PyTorch debug build False
torch._C._GLIBCXX_USE_CXX11_ABI False
GPU available Yes
GPU 0,1 NVIDIA GeForce RTX 4090 (arch=8.9)
Driver version 535.183.01
CUDA_HOME :/usr/local/cuda - invalid!
Pillow 10.3.0
torchvision 0.11.0 @/media/hky/d2bed7b9-228d-4a5b-ae76-f3f34ce12c7b/hky/anaconda3/envs/maskdino/lib/python3.8/site-packages/torchvision
torchvision arch flags /media/hky/d2bed7b9-228d-4a5b-ae76-f3f34ce12c7b/hky/anaconda3/envs/maskdino/lib/python3.8/site-packages/torchvision/_C.so; cannot find cuobjdump
fvcore 0.1.5.post20221221
iopath 0.1.9
cv2 4.10.0
------------------------------- -------------

This problem arises when I execute train_net.py:

[07/01 22:30:46] d2.engine.train_loop INFO: Starting training from iteration 0
[07/01 22:30:47] d2.engine.train_loop ERROR: Exception during training:
Traceback (most recent call last):
File "/media/hky/d2bed7b9-228d-4a5b-ae76-f3f34ce12c7b/hky/lsj/code/MASKDINO/detectron2-main/detectron2/engine/train_loop.py", line 155, in train
self.run_step()
File "/media/hky/d2bed7b9-228d-4a5b-ae76-f3f34ce12c7b/hky/lsj/code/MASKDINO/detectron2-main/detectron2/engine/defaults.py", line 498, in run_step
self._trainer.run_step()
File "/media/hky/d2bed7b9-228d-4a5b-ae76-f3f34ce12c7b/hky/lsj/code/MASKDINO/detectron2-main/detectron2/engine/train_loop.py", line 495, in run_step
loss_dict = self.model(data)
File "/media/hky/d2bed7b9-228d-4a5b-ae76-f3f34ce12c7b/hky/anaconda3/envs/maskdino/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/media/hky/d2bed7b9-228d-4a5b-ae76-f3f34ce12c7b/hky/lsj/code/MaskDINO-main/maskdino/maskdino.py", line 267, in forward
losses = self.criterion(outputs, targets,mask_dict)
File "/media/hky/d2bed7b9-228d-4a5b-ae76-f3f34ce12c7b/hky/anaconda3/envs/maskdino/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/media/hky/d2bed7b9-228d-4a5b-ae76-f3f34ce12c7b/hky/lsj/code/MaskDINO-main/maskdino/modeling/criterion.py", line 357, in forward
indices = self.matcher(outputs_without_aux, targets)
File "/media/hky/d2bed7b9-228d-4a5b-ae76-f3f34ce12c7b/hky/anaconda3/envs/maskdino/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/media/hky/d2bed7b9-228d-4a5b-ae76-f3f34ce12c7b/hky/anaconda3/envs/maskdino/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 28, in decorate_context
return func(*args, **kwargs)
File "/media/hky/d2bed7b9-228d-4a5b-ae76-f3f34ce12c7b/hky/lsj/code/MaskDINO-main/maskdino/modeling/matcher.py", line 220, in forward
return self.memory_efficient_forward(outputs, targets, cost)
File "/media/hky/d2bed7b9-228d-4a5b-ae76-f3f34ce12c7b/hky/anaconda3/envs/maskdino/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 28, in decorate_context
return func(*args, **kwargs)
File "/media/hky/d2bed7b9-228d-4a5b-ae76-f3f34ce12c7b/hky/lsj/code/MaskDINO-main/maskdino/modeling/matcher.py", line 169, in memory_efficient_forward
cost_mask = batch_sigmoid_ce_loss_jit(out_mask, tgt_mask)
RuntimeError: nvrtc: error: invalid value for --gpu-architecture (-arch)

nvrtc compilation failed:

#define NAN __int_as_float(0x7fffffff)
#define POS_INFINITY __int_as_float(0x7f800000)
#define NEG_INFINITY __int_as_float(0xff800000)

template
__device__ T maximum(T a, T b) {
return isnan(a) ? a : (a > b ? a : b);
}

template
__device__ T minimum(T a, T b) {
return isnan(a) ? a : (a < b ? a : b);
}

extern "C" __global__
void fused_neg_add(float* ttargets_1, float* aten_add) {
{
float v = __ldg(ttargets_1 + (long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x));
aten_add[(long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x)] = (0.f - v) + 1.f;
}
}

Start with the train_net.py entry point and follow the reported call path into maskdino/modeling/matcher.py, especially memory_efficient_forward. Reproduce the failure using the reported Python, PyTorch, CUDA compiler, GPU, and driver versions, then verify that training reaches this operation without the NVRTC architecture error.

Written by the indexing model from the issue text.