RuntimeError: CUDA error: unspecified launch failure

Nobody has claimed this yet.

### Bug description

The program can run normally in the debugging of vscode, and the following error will be reported when running on the command line. The format of my dataset is DGLDataset.
My pytorch version is 1.12.1+cu116
dgl version is 1.1.1+cu117
pl version is 2.0.4

```
Traceback (most recent call last):
File "/home/hmh/project/PL_survival/train.py", line 102, in
main(cfg)
File "/home/hmh/project/PL_survival/train.py", line 77, in main
trainer.fit(model = model, datamodule = dm)
File "/home/hmh/anaconda3/envs/torch112/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 531, in fit
call._call_and_handle_interrupt(
File "/home/hmh/anaconda3/envs/torch112/lib/python3.9/site-packages/pytorch_lightning/trainer/call.py", line 42, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "/home/hmh/anaconda3/envs/torch112/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 570, in _fit_impl
self._run(model, ckpt_path=ckpt_path)
File "/home/hmh/anaconda3/envs/torch112/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 975, in _run
results = self._run_stage()
File "/home/hmh/anaconda3/envs/torch112/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1018, in _run_stage
self.fit_loop.run()
File "/home/hmh/anaconda3/envs/torch112/lib/python3.9/site-packages/pytorch_lightning/loops/fit_loop.py", line 197, in run
self.on_run_start()
File "/home/hmh/anaconda3/envs/torch112/lib/python3.9/site-packages/pytorch_lightning/loops/fit_loop.py", line 308, in on_run_start
self.epoch_loop.val_loop.setup_data()
File "/home/hmh/anaconda3/envs/torch112/lib/python3.9/site-packages/pytorch_lightning/loops/evaluation_loop.py", line 173, in setup_data
length = len(dl) if has_len_all_ranks(dl, trainer.strategy, allow_zero_length) else float("inf")
File "/home/hmh/anaconda3/envs/torch112/lib/python3.9/site-packages/pytorch_lightning/utilities/data.py", line 101, in has_len_all_ranks
total_length = strategy.reduce(torch.tensor(local_length, device=strategy.root_device), reduce_op="sum")
RuntimeError: CUDA error: unspecified launch failure
```
### What version are you seeing the problem on?

v2.0

### How to reproduce the bug

_No response_

### Error messages and logs

```
# Error messages and logs here please
```

### Environment

Current environment

```
#- Lightning Component (e.g. Trainer, LightningModule, LightningApp, LightningWork, LightningFlow):
#- PyTorch Lightning Version (e.g., 1.5.0):
#- Lightning App Version (e.g., 0.5.2):
#- PyTorch Version (e.g., 2.0):
#- Python version (e.g., 3.9):
#- OS (e.g., Linux):
#- CUDA/cuDNN version:
#- GPU models and configuration:
#- How you installed Lightning(`conda`, `pip`, source):
#- Running environment of LightningApp (e.g. local, cloud):
```

### More info

_No response_

Start with train.py at lines 77 and 102, then inspect pytorch_lightning/utilities/data.py in has_len_all_ranks and loops/evaluation_loop.py during setup_data. Reproduce the command-line run with the reported versions and complete environment details; done requires a reliable reproduction and a confirmed cause or documented resolution for the CUDA launch failure.

Written by the indexing model from the issue text.