performance degradation in to_pil_image after v0.17

🐛 Describe the bug

torchvision.transforms.functional.to_pil_image is much slower when converting torch.float16 image tensors to PIL Images based on my benchmarks (serializing 360 images):

Dependencies:

Python 3.11
Pillow 10.4.0

Before (torch 2.0.1, torchvision v0.15.2, Code here): 23 seconds
After ( torch 2.2.0, torchvision v0.17, Code here): 53 seconds

How to reproduce:

import torch
from torchvision.transforms.functional import to_pil_image

rand_img_tensor = torch.rand(3, 512, 512, dtype=torch.float16)
start_time = time.time()
for _ in range(50):
    pil_img = to_pil_image(rand_img_tensor)

end_time = time.time()
print(end_time - start_time) # seconds

Run the above script with both versions of dependencies listed, and the time difference is apparent.

The cause seems to be this PR

Start in torchvision/transforms/functional.py and compare the linked v0.15.2 and v0.17 implementations, focusing on the float16 path in to_pil_image. Run the provided benchmark with the listed Python, Torch, torchvision, and Pillow versions. Done means the regression is explained and the conversion no longer shows the reported slowdown.