When calling trainer.test() train_dataloader is also validated, which makes no sense

### Bug description

In the current logic of pytorch-lightning everytime I call a` trainer.test() `it is also checked if the `train_dataloader()` function makes sense. This is problematic.

For example, I use a `WeightedRandomSampler` only in the` train_dataloader` for obvious reasons. In order for this to work I calculate
the `weights` and `num_samples` parameters in the `setup() stage="fit"` section of my code.

Of course when I trigger` trainer.test()` this code is not executed and thus weights and num_samples are never calculated, which
leads to an error when lightning validates the` train_dataloader` function.

I dont see any best practices to avoid this and no reason to validate code which is never executed.

### What version are you seeing the problem on?

v2.2

### How to reproduce the bug

_No response_

### Error messages and logs

```
# Error messages and logs here please
```

### Environment

Current environment

```
#- Lightning Component (e.g. Trainer, LightningModule, LightningApp, LightningWork, LightningFlow):
#- PyTorch Lightning Version (e.g., 1.5.0):
#- Lightning App Version (e.g., 0.5.2):
#- PyTorch Version (e.g., 2.0):
#- Python version (e.g., 3.9):
#- OS (e.g., Linux):
#- CUDA/cuDNN version:
#- GPU models and configuration:
#- How you installed Lightning(`conda`, `pip`, source):
#- Running environment of LightningApp (e.g. local, cloud):
```

### More info

_No response_

cc @justusschock @awaelchli

Start by tracing the trainer.test() entry point into validation of train_dataloader(), then inspect how setup(stage="fit") affects the WeightedRandomSampler parameters. Done means test runs without validating or requiring the unused train_dataloader() path, while fit behavior remains unchanged.

Written by the indexing model from the issue text.