Predict steps on ROCm going slowly and not recorded by profiler

Nobody has claimed this yet.

### Bug description

So, I recently changed my GPU to Radeon VII from 1050ti, and reconfigured my Conda environments to make Pytorch and Pytorch lightning run on ROCm. The process is more or less a pain, but at least the network works fine during the training of my computer vision script, although a bit slower than I would expect from a card like that.
However, after I finish training the network and starting the predict steps, I notice the script become *extremely* slow. Every predict step much longer to complete compare to the weaker 1050ti. So I enabled the profiler in pl.Trainer to see what's causing the slowness... And the answer is: the profiler has no idea.

As you can see in the image above, the whole prediction process takes around 3 minutes to complete, where predict steps only take 23 seconds in total. There is no indication of what the script is doing for the rest of the time.
This disparity increases as the size of the network increases.
[ML_halfNets.zip](https://github.com/Lightning-AI/lightning/files/11631916/ML_halfNets.zip)

### What version are you seeing the problem on?

v2.0

### How to reproduce the bug

Decompress the zip file above, then run the FlatHalfNet.py file. Observe the time taken.
Note you will need imageio and scipy packages.
You will see terminal messages similar to the log I attached below.

### Error messages and logs

There is no error message.
Log:
```
home/fgdfgfthgr/anaconda3/envs/CV/lib/python3.10/site-packages/lightning_fabric/connector.py:562: UserWarning: 16 is supported for historical reasons but its usage is discouraged. Please set your precision to 16-mixed instead!
rank_zero_warn(
Using 16bit Automatic Mixed Precision (AMP)
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
Predicting DataLoader 0: 100%|██████████| 8/8 [03:06<00:00, 23.28s/it]
PREDICT Profiler Report

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| Action | Mean duration (s) | Num calls | Total time (s) | Percentage % |
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| Total | - | 142 | 186.98 | 100 % |
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| [Strategy]SingleDeviceStrategy.predict_step | 2.9367 | 8 | 23.494 | 12.565 |
| [Strategy]SingleDeviceStrategy.batch_to_device | 0.0017125 | 8 | 0.0137 | 0.0073266 |
| [LightningModule]HalfNetPL.transfer_batch_to_device | 0.0016691 | 8 | 0.013353 | 0.0071412 |
| [_PredictionLoop].predict_next | 0.00076719 | 8 | 0.0061375 | 0.0032823 |
| [Callback]TQDMProgressBar.on_predict_start | 0.0011276 | 1 | 0.0011276 | 0.00060302 |
| [Callback]TQDMProgressBar.on_predict_batch_end | 0.0001305 | 8 | 0.001044 | 0.00055834 |
| [LightningModule]HalfNetPL.on_predict_model_eval | 0.00058303 | 1 | 0.00058303 | 0.00031181 |
| [Callback]TQDMProgressBar.on_predict_batch_start | 5.1581e-05 | 8 | 0.00041264 | 0.00022068 |
| [Callback]TQDMProgressBar.on_predict_end | 0.00017883 | 1 | 0.00017883 | 9.5636e-05 |
| [LightningModule]HalfNetPL.on_before_batch_transfer | 2.3219e-06 | 8 | 1.8575e-05 | 9.934e-06 |
| [Callback]ModelSummary.on_predict_batch_end | 1.7771e-06 | 8 | 1.4217e-05 | 7.6033e-06 |
| [Callback]ModelSummary.on_predict_batch_start | 1.6919e-06 | 8 | 1.3535e-05 | 7.2385e-06 |
| [LightningModule]HalfNetPL.on_after_batch_transfer | 1.5967e-06 | 8 | 1.2774e-05 | 6.8316e-06 |
| [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None}.setup | 1.046e-05 | 1 | 1.046e-05 | 5.594e-06 |
| [LightningModule]HalfNetPL.on_predict_batch_start | 1.2309e-06 | 8 | 9.847e-06 | 5.2662e-06 |
| [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None}.on_predict_batch_end | 1.102e-06 | 8 | 8.816e-06 | 4.7148e-06 |
| [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None}.on_predict_batch_start | 1.0181e-06 | 8 | 8.145e-06 | 4.356e-06 |
| [LightningModule]HalfNetPL.on_predict_batch_end | 9.8663e-07 | 8 | 7.893e-06 | 4.2212e-06 |
| [Callback]ModelSummary.on_predict_start | 7.424e-06 | 1 | 7.424e-06 | 3.9704e-06 |
| [Strategy]SingleDeviceStrategy.on_predict_start | 5.821e-06 | 1 | 5.821e-06 | 3.1131e-06 |
| [Callback]TQDMProgressBar.setup | 3.617e-06 | 1 | 3.617e-06 | 1.9344e-06 |
| [Callback]TQDMProgressBar.on_predict_epoch_end | 3.617e-06 | 1 | 3.617e-06 | 1.9344e-06 |
| [Callback]TQDMProgressBar.teardown | 2.876e-06 | 1 | 2.876e-06 | 1.5381e-06 |
| [LightningModule]HalfNetPL.configure_callbacks | 2.635e-06 | 1 | 2.635e-06 | 1.4092e-06 |
| [Strategy]SingleDeviceStrategy.on_predict_end | 2.294e-06 | 1 | 2.294e-06 | 1.2268e-06 |
| [LightningModule]HalfNetPL.on_predict_end | 2.144e-06 | 1 | 2.144e-06 | 1.1466e-06 |
| [Callback]ModelSummary.on_predict_end | 1.703e-06 | 1 | 1.703e-06 | 9.1077e-07 |
| [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None}.on_predict_start | 1.623e-06 | 1 | 1.623e-06 | 8.6798e-07 |
| [Callback]TQDMProgressBar.on_predict_epoch_start | 1.573e-06 | 1 | 1.573e-06 | 8.4124e-07 |
| [Callback]ModelSummary.setup | 1.433e-06 | 1 | 1.433e-06 | 7.6637e-07 |
| [LightningModule]HalfNetPL.on_predict_start | 1.433e-06 | 1 | 1.433e-06 | 7.6637e-07 |
| [Callback]ModelSummary.teardown | 1.202e-06 | 1 | 1.202e-06 | 6.4283e-07 |
| [LightningModule]HalfNetPL.teardown | 1.182e-06 | 1 | 1.182e-06 | 6.3214e-07 |
| [LightningModule]HalfNetPL.prepare_data | 1.162e-06 | 1 | 1.162e-06 | 6.2144e-07 |
| [LightningModule]HalfNetPL.setup | 1.152e-06 | 1 | 1.152e-06 | 6.1609e-07 |
| [LightningModule]HalfNetPL.configure_sharded_model | 1.122e-06 | 1 | 1.122e-06 | 6.0005e-07 |
| [Callback]ModelSummary.on_predict_epoch_end | 1.082e-06 | 1 | 1.082e-06 | 5.7866e-07 |
| [LightningModule]HalfNetPL.on_predict_epoch_end | 1.082e-06 | 1 | 1.082e-06 | 5.7866e-07 |
| [Callback]ModelSummary.on_predict_epoch_start | 1.052e-06 | 1 | 1.052e-06 | 5.6261e-07 |
| [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None}.on_predict_epoch_start | 1.032e-06 | 1 | 1.032e-06 | 5.5192e-07 |
| [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None}.teardown | 9.52e-07 | 1 | 9.52e-07 | 5.0913e-07 |
| [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None}.on_predict_epoch_end | 9.51e-07 | 1 | 9.51e-07 | 5.086e-07 |
| [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None}.on_predict_end | 9.12e-07 | 1 | 9.12e-07 | 4.8774e-07 |
| [LightningModule]HalfNetPL.on_predict_epoch_start | 8.41e-07 | 1 | 8.41e-07 | 4.4977e-07 |
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

进程已结束,退出代码0

```

### Environment

Current environment

* CUDA:
- GPU:
- AMD Radeon VII
- available: True
- version: None
* Lightning:
- lightning-utilities: 0.8.0
- pytorch-lightning: 2.0.2
- pytorch-triton-rocm: 2.0.1
- torch: 2.0.1+rocm5.4.2
- torchaudio: 2.0.2+rocm5.4.2
- torchmetrics: 0.11.4
- torchvision: 0.15.2+rocm5.4.2
* Packages:
- absl-py: 1.4.0
- aiohttp: 3.8.4
- aiosignal: 1.3.1
- async-timeout: 4.0.2
- attrs: 23.1.0
- cachetools: 5.3.1
- certifi: 2022.12.7
- charset-normalizer: 2.1.1
- cmake: 3.25.0
- filelock: 3.9.0
- frozenlist: 1.3.3
- fsspec: 2023.5.0
- google-auth: 2.19.1
- google-auth-oauthlib: 1.0.0
- grpcio: 1.54.2
- idna: 3.4
- imageio: 2.30.0
- jinja2: 3.1.2
- lightning-utilities: 0.8.0
- lit: 15.0.7
- markdown: 3.4.3
- markupsafe: 2.1.2
- mpmath: 1.2.1
- multidict: 6.0.4
- networkx: 3.0
- numpy: 1.24.1
- oauthlib: 3.2.2
- packaging: 23.1
- pillow: 9.3.0
- pip: 23.0.1
- protobuf: 4.23.2
- pyasn1: 0.5.0
- pyasn1-modules: 0.3.0
- pytorch-lightning: 2.0.2
- pytorch-triton-rocm: 2.0.1
- pyyaml: 6.0
- requests: 2.28.1
- requests-oauthlib: 1.3.1
- rsa: 4.9
- scipy: 1.10.1
- setuptools: 67.8.0
- six: 1.16.0
- sympy: 1.11.1
- tensorboard: 2.13.0
- tensorboard-data-server: 0.7.0
- torch: 2.0.1+rocm5.4.2
- torchaudio: 2.0.2+rocm5.4.2
- torchmetrics: 0.11.4
- torchvision: 0.15.2+rocm5.4.2
- tqdm: 4.65.0
- typing-extensions: 4.4.0
- urllib3: 1.26.13
- werkzeug: 2.3.4
- wheel: 0.38.4
- yarl: 1.9.2
* System:
- OS: Linux
- architecture:
- 64bit
- ELF
- processor: x86_64
- python: 3.10.11
- release: 5.15.0-73-generic
- version: #80-Ubuntu SMP Mon May 15 15:18:26 UTC 2023

### More info

_No response_

Decompress ML_halfNets.zip and run FlatHalfNet.py with the reported ROCm environment, imageio, and scipy dependencies. Compare the profiler's total prediction time with the recorded predict_step time and determine which unreported work accounts for the gap; done means the missing time is identified and the prediction profiler reports it appropriately.

Written by the indexing model from the issue text.