lightning version is the SLURM job number when run on a node provisioned by SLURM

Nobody has claimed this yet.

### Bug description

When I use lightning on my personal computer the logs are named with an ascending version number. When I use lightning on a cluster where access is provisioned via SLURM, the version number is the job id number of the SLURM job:

```
Epoch 9: 100%|█| 166/166 [01:25<00:00, 1.95it/s, loss=0.944, v_num=8483572, train_loss_step=0.925, val_loss=0.896...
```

In the above line v_num=8483572 is the SLURM job number. The logs are also saved in a directory named `version_8483572`.

I don't know if this is working as intended.

### What version are you seeing the problem on?

v1.9

### How to reproduce the bug

```python
The following script from pytorch-forecasting will reproduce the possible bug on the cluster I work on:

import sys

import pandas as pd
import pytorch_lightning as pl
from pytorch_lightning.callbacks import EarlyStopping
from sklearn.preprocessing import scale

from pytorch_forecasting import NBeats, TimeSeriesDataSet
from pytorch_forecasting.data import NaNLabelEncoder
from pytorch_forecasting.data.examples import generate_ar_data

sys.path.append("..")

print("load data")
data = generate_ar_data(seasonality=10.0, timesteps=400, n_series=100)
data["static"] = 2
data["date"] = pd.Timestamp("2020-01-01") + pd.to_timedelta(data.time_idx, "D")
# validation = data.series.sample(20)

max_encoder_length = 150
max_prediction_length = 20

training_cutoff = data["time_idx"].max() - max_prediction_length

context_length = max_encoder_length
prediction_length = max_prediction_length

training = TimeSeriesDataSet(
data[lambda x: x.time_idx < training_cutoff],
time_idx="time_idx",
target="value",
categorical_encoders={"series": NaNLabelEncoder().fit(data.series)},
group_ids=["series"],
min_encoder_length=context_length,
max_encoder_length=context_length,
max_prediction_length=prediction_length,
min_prediction_length=prediction_length,
time_varying_unknown_reals=["value"],
randomize_length=None,
add_relative_time_idx=False,
add_target_scales=False,
)

validation = TimeSeriesDataSet.from_dataset(training, data,
min_prediction_idx=training_cutoff)
batch_size = 128
train_dataloader = training.to_dataloader(train=True, batch_size=batch_size,
num_workers=5)
val_dataloader = validation.to_dataloader(train=False, batch_size=batch_size,
num_workers=5)

early_stop_callback = EarlyStopping(monitor="val_loss", min_delta=1e-4,
patience=10, verbose=False, mode="min")
trainer = pl.Trainer(
max_epochs=10,
accelerator='cpu',
devices=1,
gradient_clip_val=0.1,
callbacks=[early_stop_callback],
limit_train_batches=1.0,
log_every_n_steps=1,
# limit_val_batches=1,
# fast_dev_run=True,
# logger=logger,
# profiler=True,
)

net = NBeats.from_dataset(
training, learning_rate=3e-2, log_interval=10, log_val_interval=1, log_gradient_flow=False, weight_decay=1e-2
)
print(f"Number of parameters in network: {net.size()/1e3:.1f}k")

# # find optimal learning rate
# # remove logging and artificial epoch size
# net.hparams.log_interval = -1
# net.hparams.log_val_interval = -1
# trainer.limit_train_batches = 1.0
# # run learning rate finder
# res = trainer.tuner.lr_find(
# net, train_dataloaders=train_dataloader, val_dataloaders=val_dataloader, min_lr=1e-5, max_lr=1e2
# )
# print(f"suggested learning rate: {res.suggestion()}")
# fig = res.plot(show=True, suggest=True)
# fig.show()
# net.hparams.learning_rate = res.suggestion()

trainer.fit(
net,
train_dataloaders=train_dataloader,
val_dataloaders=val_dataloader,
)
```
```

### Error messages and logs

```
# Error messages and logs here please
```

### Environment

Current environment

```
#- Lightning Component (e.g. Trainer, LightningModule, LightningApp, LightningWork, LightningFlow):
#- PyTorch Lightning Version (e.g., 1.5.0):
#- Lightning App Version (e.g., 0.5.2):
#- PyTorch Version (e.g., 2.0):
#- Python version (e.g., 3.9):
#- OS (e.g., Linux):
#- CUDA/cuDNN version:
#- GPU models and configuration:
#- How you installed Lightning(`conda`, `pip`, source):
#- Running environment of LightningApp (e.g. local, cloud):
```

### More info

_No response_

cc @awaelchli

Start with the provided pytorch-forecasting script and its Trainer.fit entry point, running it locally and on a SLURM-provisioned node to compare logger versions and output directories. Trace how the Trainer creates the logger and version value; done means determining whether the SLURM job ID is intentional and defining the expected versioning behavior.

Written by the indexing model from the issue text.