RuntimeError: d.is_cuda() INTERNAL ASSERT FAILED

Every time I train Mask2Former with Detectron2, I encounter the same error. The specific error is as follows:

Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
RuntimeError: d.is_cuda() INTERNAL ASSERT FAILED at "/opt/conda/conda-bld/pytorch_1678402421473/work/c10/cuda/impl/CUDAGuardImpl.h":31, please report a bug to PyTorch.

This error does not appear at the beginning but usually occurs after running for a long time, such as 5 hours. This makes debugging very difficult. I would be very grateful if someone could help me look into this issue.