XPU out of memory when upscaling image using model with novram option

### Expected Behavior

upscaling image (using model) works with novram option

### Actual Behavior

upscaling image (using model) failed and reports "XPU out of memory" even with novram option

### Steps to Reproduce

1. launch ComfyUI with python main.py --novram
2. create a new workflow with default nodes
3. add upscale (using model) nodes
4. run workflow

### Debug Logs

```text
lialh4@lialh4-pc /m/h/l/ComfyUI (master)> ./start.sh (base)

:: initializing oneAPI environment ...
start.sh: BASH_VERSION = 5.2.32(1)-release
args: Using "$@" for setvars.sh arguments:
:: advisor -- latest
:: ccl -- latest
:: compiler -- latest
:: dal -- latest
:: debugger -- latest
:: dev-utilities -- latest
:: dnnl -- latest
:: dpcpp-ct -- latest
:: dpl -- latest
:: ipp -- latest
:: ippcp -- latest
:: mkl -- latest
:: mpi -- latest
:: pti -- latest
:: tbb -- latest
:: umf -- latest
:: vtune -- latest
:: oneAPI environment initialized ::

[START] Security scan
[DONE] Security scan
## ComfyUI-Manager: installing dependencies done.
** ComfyUI startup time: 2025-01-07 20:23:43.041368
** Platform: Linux
** Python version: 3.12.8 | packaged by Anaconda, Inc. | (main, Dec 11 2024, 16:31:09) [GCC 11.2.0]
** Python executable: /media/hdd/lialh4/Conda/envs/comfyui/bin/python
** ComfyUI Path: /media/hdd/lialh4/ComfyUI
** Log path: /media/hdd/lialh4/ComfyUI/user/comfyui.log

Prestartup times for custom nodes:
5.9 seconds: /media/hdd/lialh4/ComfyUI/custom_nodes/comfyui-manager

/media/hdd/lialh4/Conda/envs/comfyui/lib/python3.12/site-packages/torchvision/io/image.py:14: UserWarning: Failed to load image Python extension: 'libjpeg.so.8: cannot open shared object file: No such file or directory'If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?
warn(
[W107 20:24:20.864104551 OperatorEntry.cpp:155] Warning: Warning only once for all operators, other operators may also be overridden.
Overriding a previously registered kernel for the same operator and the same dispatch key
operator: aten::_cummax_helper(Tensor self, Tensor(a!) values, Tensor(b!) indices, int dim) -> ()
registered at /build/pytorch/build/aten/src/ATen/RegisterSchema.cpp:6
dispatch key: XPU
previous kernel: registered at /build/pytorch/build/aten/src/ATen/RegisterCPU.cpp:30476
new kernel: registered at /build/intel-pytorch-extension/build/Release/csrc/gpu/csrc/aten/generated/ATen/RegisterXPU.cpp:2971 (function operator())
Total VRAM 5784 MB, total RAM 15843 MB
pytorch version: 2.5.1+cxx11.abi
Set vram state to: NO_VRAM
Device: xpu
Using pytorch attention
[Prompt Server] web root: /media/hdd/lialh4/ComfyUI/web
### Loading: ComfyUI-Manager (V3.3.13)
### ComfyUI Version: v0.3.10-38-g4209edf | Released on '2025-01-07'

Import times for custom nodes:
0.0 seconds: /media/hdd/lialh4/ComfyUI/custom_nodes/websocket_image_save.py
0.2 seconds: /media/hdd/lialh4/ComfyUI/custom_nodes/comfyui-manager

Starting server

To see the GUI go to: http://127.0.0.1:8188
[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/alter-list.json
[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/model-list.json
[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/custom-node-list.json
[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/github-stats.json
[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/extension-node-map.json
got prompt
model weight dtype torch.float16, manual cast: None
model_type EPS
Using pytorch attention in VAE
Using pytorch attention in VAE
VAE load device: xpu:0, offload device: cpu, dtype: torch.bfloat16
Requested to load SD1ClipModel
loaded completely 9.5367431640625e+25 235.84423828125 True
2025-01-07 20:25:03,436 - _logger.py - IPEX - INFO - Conv BatchNorm folding failed during the optimize process.
2025-01-07 20:25:03,443 - _logger.py - IPEX - INFO - Linear BatchNorm folding failed during the optimize process.
CLIP model load device: cpu, offload device: cpu, current: cpu, dtype: torch.float16
2025-01-07 20:25:05,020 - _logger.py - IPEX - INFO - Conv BatchNorm folding failed during the optimize process.
2025-01-07 20:25:05,025 - _logger.py - IPEX - INFO - Linear BatchNorm folding failed during the optimize process.
2025-01-07 20:25:08,997 - _logger.py - IPEX - INFO - Conv BatchNorm folding failed during the optimize process.
2025-01-07 20:25:09,006 - _logger.py - IPEX - INFO - Linear BatchNorm folding failed during the optimize process.
Requested to load BaseModel
loaded partially 9.5367431640625e-08 0.0 0
2025-01-07 20:25:12,921 - _logger.py - IPEX - INFO - Conv BatchNorm folding failed during the optimize process.
2025-01-07 20:25:12,931 - _logger.py - IPEX - INFO - Linear BatchNorm folding failed during the optimize process.
0%| | 0/20 [00:00
s = comfy.utils.tiled_scale(in_img, lambda a: upscale_model(a), tile_x=tile, tile_y=tile, overlap=overlap, upscale_amount=upscale_model.scale, pbar=pbar)
^^^^^^^^^^^^^^^^
File "/media/hdd/lialh4/Conda/envs/comfyui/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/media/hdd/lialh4/Conda/envs/comfyui/lib/python3.12/site-packages/spandrel/__helpers/model_descriptor.py", line 472, in __call__
output = self._call_fn(self.model, image)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/media/hdd/lialh4/Conda/envs/comfyui/lib/python3.12/site-packages/spandrel/__helpers/model_descriptor.py", line 439, in
self._call_fn = call_fn or (lambda model, image: model(image))
^^^^^^^^^^^^
File "/media/hdd/lialh4/Conda/envs/comfyui/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/media/hdd/lialh4/Conda/envs/comfyui/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/media/hdd/lialh4/Conda/envs/comfyui/lib/python3.12/site-packages/spandrel/architectures/ESRGAN/__arch/RRDB.py", line 143, in forward
return self.model(x)
^^^^^^^^^^^^^
File "/media/hdd/lialh4/Conda/envs/comfyui/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/media/hdd/lialh4/Conda/envs/comfyui/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/media/hdd/lialh4/Conda/envs/comfyui/lib/python3.12/site-packages/torch/nn/modules/container.py", line 250, in forward
input = module(input)
^^^^^^^^^^^^^
File "/media/hdd/lialh4/Conda/envs/comfyui/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/media/hdd/lialh4/Conda/envs/comfyui/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/media/hdd/lialh4/Conda/envs/comfyui/lib/python3.12/site-packages/torch/nn/modules/conv.py", line 554, in forward
return self._conv_forward(input, self.weight, self.bias)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/media/hdd/lialh4/Conda/envs/comfyui/lib/python3.12/site-packages/torch/nn/modules/conv.py", line 549, in _conv_forward
return F.conv2d(
^^^^^^^^^
RuntimeError: XPU out of memory. Tried to allocate 2.00 GiB (GPU 0; 5.65 GiB total capacity; 2.07 GiB already allocated; 4.24 GiB reserved in total by PyTorch)

Prompt executed in 137.88 seconds
```

### Other

GPU: Intel Arc A380 (6GB VRAM)

error screenshot:

workflow screenshot:

generate image using default workflow at 768x768 with novram option gives same "XPU out of memory" error, maybe it's IPEX related bug?

start.sh content:
```sh
#!/bin/sh
source /opt/intel/oneapi/setvars.sh
conda run --live-stream -n comfyui python main.py --novram #--oneapi-device-selector '*:0'
```