Resume from checkpoint not working on multi-gpu instance

Nobody has claimed this yet.

### Bug description

I have trained a model using multiple gpu . after that i am trying to resume training from checkpoint on the same multi-gpu instance but i am getting memory related error.
but same when i am trying on single gpu it started training but its not started from last epoch i guess becuase loss if high this time .
also does it use the same learning rate that was found in previous training usnig lr_finder callback.?

### What version are you seeing the problem on?

1.9.x

### How to reproduce the bug

```python
trainer = pl.Trainer(logger=wandb_logger,resume_from_checkpoint="./580sdysk/checkpoints/model-epoch=09-val_loss=0.01.ckpt",
accelerator=config.ACCELERATOR,
devices=config.DEVICES,
min_epochs=1,
max_epochs=config.NUM_EPOCHS,
callbacks=[checkpoint_callback,lr_findr]
)

trainer.fit(model, dm)#,ckpt_path="./580sdysk/checkpoints/model-epoch=09-val_loss=0.01.ckpt")
trainer.validate(model, dm)
trainer.test(model, dm)
```

### Error messages and logs

# Error messages and logs here please
```
Traceback (most recent call last):
File "train.py", line 70, in
trainer.fit(model, dm)#,ckpt_path="./emb_training/580sdysk/checkpoints/Content_Emb-epoch=09-val_loss=0.01.ckpt")
File "/opt/conda/envs/emb_poc_v1/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 608, in fit
call._call_and_handle_interrupt(
File "/opt/conda/envs/emb_poc_v1/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 36, in _call_and_handle_interrupt
return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
File "/opt/conda/envs/emb_poc_v1/lib/python3.8/site-packages/pytorch_lightning/strategies/launchers/multiprocessing.py", line 113, in launch
mp.start_processes(
File "/opt/conda/envs/emb_poc_v1/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 197, in start_processes
while not context.join():
File "/opt/conda/envs/emb_poc_v1/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 140, in join
raise ProcessExitedException(
torch.multiprocessing.spawn.ProcessExitedException: process 1 terminated with signal SIGKILL
```
### Environment

Current environment

* CUDA:
- GPU:
- Tesla T4
- Tesla T4
- Tesla T4
- Tesla T4
- available: True
- version: 11.7
* Lightning:
- lightning-utilities: 0.8.0
- pytorch-lightning: 1.9.4
- torch: 2.0.0+cu117
- torchaudio: 2.0.1+cu117
- torchmetrics: 0.11.4
- torchvision: 0.15.1+cu117
* Packages:
- absl-py: 1.4.0
- aiohttp: 3.8.4
- aiosignal: 1.3.1
- anyio: 3.6.2
- appdirs: 1.4.4
- argon2-cffi: 21.3.0
- argon2-cffi-bindings: 21.2.0
- arrow: 1.2.3
- asttokens: 2.2.1
- async-timeout: 4.0.2
- attrs: 23.1.0
- backcall: 0.2.0
- beautifulsoup4: 4.12.2
- bleach: 6.0.0
- cachetools: 5.3.0
- certifi: 2022.12.7
- cffi: 1.15.1
- charset-normalizer: 2.1.1
- click: 8.1.3
- cmake: 3.25.0
- comm: 0.1.3
- debugpy: 1.6.7
- decorator: 5.1.1
- defusedxml: 0.7.1
- docker-pycreds: 0.4.0
- executing: 1.2.0
- fastjsonschema: 2.16.3
- filelock: 3.9.0
- fqdn: 1.5.1
- frozenlist: 1.3.3
- fsspec: 2023.3.0
- gcsfs: 2023.3.0
- gitdb: 4.0.10
- gitpython: 3.1.31
- google-api-core: 2.11.0
- google-auth: 2.17.1
- google-auth-oauthlib: 1.0.0
- google-cloud-core: 2.3.2
- google-cloud-storage: 2.8.0
- google-crc32c: 1.5.0
- google-resumable-media: 2.4.1
- googleapis-common-protos: 1.59.0
- grpcio: 1.53.0
- huggingface-hub: 0.13.3
- idna: 3.4
- importlib-metadata: 6.5.0
- importlib-resources: 5.12.0
- ipykernel: 6.22.0
- ipython: 8.12.0
- ipython-genutils: 0.2.0
- ipywidgets: 8.0.6
- isoduration: 20.11.0
- jedi: 0.18.2
- jinja2: 3.1.2
- joblib: 1.2.0
- jsonpointer: 2.3
- jsonschema: 4.17.3
- jupyter: 1.0.0
- jupyter-client: 8.2.0
- jupyter-console: 6.6.3
- jupyter-core: 5.3.0
- jupyter-events: 0.6.3
- jupyter-server: 2.5.0
- jupyter-server-terminals: 0.4.4
- jupyterlab-pygments: 0.2.2
- jupyterlab-widgets: 3.0.7
- lightning-utilities: 0.8.0
- lit: 15.0.7
- markdown: 3.4.3
- markupsafe: 2.1.2
- matplotlib-inline: 0.1.6
- mistune: 2.0.5
- mpmath: 1.2.1
- multidict: 6.0.4
- nbclassic: 0.5.5
- nbclient: 0.7.3
- nbconvert: 7.3.1
- nbformat: 5.8.0
- nest-asyncio: 1.5.6
- networkx: 3.0
- notebook: 6.5.4
- notebook-shim: 0.2.2
- numpy: 1.24.1
- oauthlib: 3.2.2
- openai: 0.27.4
- packaging: 23.1
- pandas: 2.0.0
- pandocfilters: 1.5.0
- parso: 0.8.3
- pathtools: 0.1.2
- pexpect: 4.8.0
- pickleshare: 0.7.5
- pillow: 9.3.0
- pip: 23.0.1
- pkgutil-resolve-name: 1.3.10
- platformdirs: 3.2.0
- prometheus-client: 0.16.0
- prompt-toolkit: 3.0.38
- protobuf: 4.22.1
- psutil: 5.9.5
- ptyprocess: 0.7.0
- pure-eval: 0.2.2
- pyasn1: 0.4.8
- pyasn1-modules: 0.2.8
- pycparser: 2.21
- pygments: 2.15.1
- pyrsistent: 0.19.3
- python-dateutil: 2.8.2
- python-dotenv: 1.0.0
- python-json-logger: 2.0.7
- pytorch-lightning: 1.9.4
- pytz: 2023.3
- pyyaml: 6.0
- pyzmq: 25.0.2
- qtconsole: 5.4.2
- qtpy: 2.3.1
- regex: 2023.3.23
- requests: 2.28.1
- requests-oauthlib: 1.3.1
- rfc3339-validator: 0.1.4
- rfc3986-validator: 0.1.1
- rsa: 4.9
- scikit-learn: 0.23.0
- scipy: 1.10.1
- send2trash: 1.8.0
- sentry-sdk: 1.19.0
- setproctitle: 1.3.2
- setuptools: 67.6.1
- six: 1.16.0
- smmap: 5.0.0
- sniffio: 1.3.0
- soupsieve: 2.4.1
- stack-data: 0.6.2
- sympy: 1.11.1
- tensorboard: 2.12.2
- tensorboard-data-server: 0.7.0
- tensorboard-plugin-wit: 1.8.1
- terminado: 0.17.1
- threadpoolctl: 3.1.0
- tiktoken: 0.3.3
- tinycss2: 1.2.1
- tokenizers: 0.13.2
- torch: 2.0.0+cu117
- torchaudio: 2.0.1+cu117
- torchmetrics: 0.11.4
- torchvision: 0.15.1+cu117
- tornado: 6.3
- tqdm: 4.65.0
- traitlets: 5.9.0
- transformers: 4.27.4
- triton: 2.0.0
- typing-extensions: 4.5.0
- tzdata: 2023.3
- uri-template: 1.2.0
- urllib3: 1.26.13
- wandb: 0.14.0
- wcwidth: 0.2.6
- webcolors: 1.13
- webencodings: 0.5.1
- websocket-client: 1.5.1
- werkzeug: 2.2.3
- wheel: 0.40.0
- widgetsnbextension: 4.0.7
- yarl: 1.8.2
- zipp: 3.15.0
* System:
- OS: Linux
- architecture:
- 64bit
- ELF
- processor:
- python: 3.8.13
- version: #1 SMP Debian 4.19.269-1 (2022-12-20)

### More info

how can i get the values of learning rate found using lr_finder callback

Start at trainer.fit with resume_from_checkpoint and the multiprocessing launcher shown in the traceback, then inspect how the lr_findr callback participates in resumed training. Reproduce the four-GPU and single-GPU cases with the provided configuration; done should establish whether checkpoint state and the lr_finder learning rate are restored consistently and identify the source of the SIGKILL.

Written by the indexing model from the issue text.