Upscale image (via model) unusably slow when using loaded image

The Upscale image (via model) node works perfectly if I connect its image input to the output of a VAE decode (which is the last step of a txt2img workflow).

However, I want a workflow for upscaling images that I have generated previously. To do this, I removed the txt2img workflow, and simply pass an image from the Load image node to the Upscale image node.

However, when running this, it seems abnormally slow. There is no progress at all, ComfyUI starts hogging 1 CPU core 100%, and my computer becomes unusably slow (to the point of freezing). This has to be abnormal.

I don't understand however why this works when preceded by a text2img workflow, but not when loading an image from file.