Fabric dataloaders die when num workers > 0 and distributed.

Nobody has claimed this yet.

### Bug description

When using fabric for distributed training using ddp or fsdp as strategy on two 1080ti or 2080ti, line of code:

```
for batch_idx, batch in enumerate(train_dataloader):
```

causes errors after a few epochs.

Full traceback:

```
terminate called after throwing an instance of 'c10::Error'
what(): CUDA error: initialization error
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:44 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f4c648004d7 in /home/miles/anaconda3/envs/pl2/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7f4c647ca36b in /home/miles/anaconda3/envs/pl2/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7f4c648a4fa8 in /home/miles/anaconda3/envs/pl2/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: + 0x135fe (0x7f4c648755fe in /home/miles/anaconda3/envs/pl2/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #4: + 0x22d80 (0x7f4c64884d80 in /home/miles/anaconda3/envs/pl2/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #5: + 0x4ccea6 (0x7f4ca3d37ea6 in /home/miles/anaconda3/envs/pl2/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #6: + 0x3ee77 (0x7f4c647e5e77 in /home/miles/anaconda3/envs/pl2/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #7: c10::TensorImpl::~TensorImpl() + 0x1be (0x7f4c647de69e in /home/miles/anaconda3/envs/pl2/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #8: c10::TensorImpl::~TensorImpl() + 0x9 (0x7f4c647de7b9 in /home/miles/anaconda3/envs/pl2/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #9: + 0x752458 (0x7f4ca3fbd458 in /home/miles/anaconda3/envs/pl2/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #10: THPVariable_subclass_dealloc(_object*) + 0x305 (0x7f4ca3fbd7e5 in /home/miles/anaconda3/envs/pl2/lib/python3.10/site-packages/torch/lib/libtorch_python.so)

Traceback (most recent call last):
File "/home/miles/anaconda3/envs/pl2/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1133, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/home/miles/anaconda3/envs/pl2/lib/python3.10/multiprocessing/queues.py", line 122, in get
return _ForkingPickler.loads(res)
File "/home/miles/anaconda3/envs/pl2/lib/python3.10/site-packages/torch/multiprocessing/reductions.py", line 307, in rebuild_storage_fd
fd = df.detach()
File "/home/miles/anaconda3/envs/pl2/lib/python3.10/multiprocessing/resource_sharer.py", line 57, in detach
with _resource_sharer.get_connection(self._id) as conn:
File "/home/miles/anaconda3/envs/pl2/lib/python3.10/multiprocessing/resource_sharer.py", line 86, in get_connection
c = Client(address, authkey=process.current_process().authkey)
File "/home/miles/anaconda3/envs/pl2/lib/python3.10/multiprocessing/connection.py", line 508, in Client
answer_challenge(c, authkey)
File "/home/miles/anaconda3/envs/pl2/lib/python3.10/multiprocessing/connection.py", line 752, in answer_challenge
message = connection.recv_bytes(256) # reject large message
File "/home/miles/anaconda3/envs/pl2/lib/python3.10/multiprocessing/connection.py", line 216, in recv_bytes
buf = self._recv_bytes(maxlength)
File "/home/miles/anaconda3/envs/pl2/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
buf = self._recv(4)
File "/home/miles/anaconda3/envs/pl2/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
chunk = read(handle, remaining)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/miles/Programming/pl2f_error/src/main.py", line 137, in
main(args)
File "/home/miles/Programming/pl2f_error/src/main.py", line 54, in main
train(fabric, model, optimizer, train_dataloader, val_dataloader)
File "/home/miles/Programming/pl2f_error/src/main.py", line 73, in train
for batch_idx, batch in enumerate(train_dataloader):
File "/home/miles/anaconda3/envs/pl2/lib/python3.10/site-packages/lightning/fabric/wrappers.py", line 178, in __iter__
for item in self._dataloader:
File "/home/miles/anaconda3/envs/pl2/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 634, in __next__
data = self._next_data()
File "/home/miles/anaconda3/envs/pl2/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1329, in _next_data
idx, data = self._get_data()
File "/home/miles/anaconda3/envs/pl2/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1295, in _get_data
success, data = self._try_get_data()
File "/home/miles/anaconda3/envs/pl2/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1157, in _try_get_data
fs = [tempfile.NamedTemporaryFile() for i in range(fds_limit_margin)]
File "/home/miles/anaconda3/envs/pl2/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1157, in
fs = [tempfile.NamedTemporaryFile() for i in range(fds_limit_margin)]
File "/home/miles/anaconda3/envs/pl2/lib/python3.10/tempfile.py", line 559, in NamedTemporaryFile
file = _io.open(dir, mode, buffering=buffering,
File "/home/miles/anaconda3/envs/pl2/lib/python3.10/tempfile.py", line 556, in opener
fd, name = _mkstemp_inner(dir, prefix, suffix, flags, output_type)
File "/home/miles/anaconda3/envs/pl2/lib/python3.10/tempfile.py", line 256, in _mkstemp_inner
fd = _os.open(file, flags, 0o600)
File "/home/miles/anaconda3/envs/pl2/lib/python3.10/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 727166) is killed by signal: Aborted.
```

Minimal repo to reproduce issue: https://github.com/mileseverett/pl2f_error

By setting num_workers to 0 in main.py code will run fine, any num_workers > 0 has caused issues for me.

### What version are you seeing the problem on?

2.0+

### How to reproduce the bug

_No response_

### Error messages and logs

```
# Error messages and logs here please
```

### Environment

Current environment

```
#- Lightning Component (e.g. Trainer, LightningModule, LightningApp, LightningWork, LightningFlow):
#- PyTorch Lightning Version (e.g., 1.5.0):
#- Lightning App Version (e.g., 0.5.2):
#- PyTorch Version (e.g., 2.0):
#- Python version (e.g., 3.9):
#- OS (e.g., Linux):
#- CUDA/cuDNN version:
#- GPU models and configuration:
#- How you installed Lightning(`conda`, `pip`, source):
#- Running environment of LightningApp (e.g. local, cloud):
```

### More info

_No response_

cc @justusschock @awaelchli @carmocca

Start with the linked minimal repository and main.py, reproducing the distributed Fabric run with num_workers greater than 0 and comparing it with num_workers set to 0. Trace the train_dataloader iteration and worker failure shown in the traceback; done means distributed training runs without workers being aborted.

Written by the indexing model from the issue text.