load_from_checkpoint Right After fit Got FileNotFound Error

Nobody has claimed this yet.

### Bug description

Hi team,

I was testing codes on 2 GPUs and was using ddp. I immediately called `MyModel.load_from_checkpoint(trainer.checkpoint_callback.best_model_path)` after `trainer.fit()` and the max_epoch is 1. However, I got FileNotFound error.

I'm actually not sure if it was caused by that trainer.fit() ends before saving checkpoint or ddp. Can you please help me figure this out?

Here is another issue #5512 which has been labeled as fixed that is related to mine

### What version are you seeing the problem on?

v1.9

### How to reproduce the bug

```python
trainer.fit(model, train_dataloader, valid_dataloader)
logger.info(f"Best model loaded from {trainer.checkpoint_callback.best_model_path}")
model = BaseModel.load_from_checkpoint(trainer.checkpoint_callback.best_model_path)
```

### Error messages and logs

```
(log) Best model loaded from /home/.../checkpoint.ckpt
...
FileNotFoundError: [Errno 2] No such file or directory: '/home/.../checkpoint.ckpt'
```

### Environment

GPU:
```
NVIDIA GeForce RTX 2080 Ti
NVIDIA TITAN RTX
```

Packages:
```
# Name Version Build Channel
pytorch 1.13.1 py3.10_cuda11.7_cudnn8.5.0_0 pytorch
pytorch-cuda 11.7 h778d358_3 pytorch
pytorch-lightning 1.9.4 pyhd8ed1ab_1 conda-forge
torchmetrics 0.10.3 pyhd8ed1ab_0 conda-forge
torchtext 0.14.1 py310 pytorch
```

### More info

_No response_

Start by reproducing the reported trainer.fit() followed by load_from_checkpoint() with two GPUs, DDP, and max_epoch=1, using the logged best_model_path. Read the checkpoint callback behavior around saving and the related issue #5512; done means the checkpoint path produced after fit consistently exists and can be loaded.

Written by the indexing model from the issue text.