Comfy-Org / Comfy-Org/ComfyUI

SeedVR2 -- Obscene memory use on MPS during tiled VAE encode/decode

Open
#15,053 1 comment 1 reaction 0 assignees View on GitHub
Potential Bug
Dominant language
Python
Stars
133k
Forks
15.7k
Avg merge
1d 7h
Merged PRs (30d)
158

Description

### Custom Node Testing

- [x] I have tried disabling custom nodes and the issue persists (see [how to disable custom nodes](https://docs.comfy.org/troubleshooting/custom-node-issues#step-1%3A-test-with-all-custom-nodes-disabled) if you need help)

### Expected Behavior

A minimal workflow consisting of importing a 1MP image, naively resizing it by 3x, and doing SeedVR2 3B fp16 reconstruction on it should produce valid output in reasonable time using reasonable memory.

### Actual Behavior

During tiled VAE encode (tested 1024 and 512 tiles), Python memory use climbs to ~73 GB, but succeeds with a lot of swapping. Memory use falls to ~15 GB during sampling, but then climbs during tiled VAE decode to 88 GB and fails with an MPS out of memory error.

### Steps to Reproduce

Import the attached workflow. Load a 1 MP image of your choice. I used a 1344x768 image. Run with batch count of 1. [SeedVR2 3B fp16 minimal workflow.json](https://github.com/user-attachments/files/30316922/SeedVR2.3B.fp16.minimal.workflow.json)

### Debug Logs

```powershell
Command: ['/Users/adreitz/pinokio/api/ComfyUI/app/env/bin/python', 'main.py', '--cache-none', '--use-quad-cross-attention']
/Users/adreitz/pinokio/bin/miniconda/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
[INFO] setup plugin alembic.autogenerate.schemas
[INFO] setup plugin alembic.autogenerate.tables
[INFO] setup plugin alembic.autogenerate.types
[INFO] setup plugin alembic.autogenerate.constraints
[INFO] setup plugin alembic.autogenerate.defaults
[INFO] setup plugin alembic.autogenerate.comments
[WARNING] Could not autodetect AIMDO implementation, assuming Nvidia
[INFO] comfy-aimdo unsupported operating system: Darwin
[INFO] NOTE: comfy-aimdo currently only supports Windows and Linux
[START] Security scan
[DONE] Security scan
## ComfyUI-Manager: installing dependencies done.
** ComfyUI startup time: 2026-07-23 13:25:43.511
** Platform: Darwin
** Python version: 3.10.20 | packaged by conda-forge | (main, Jun 11 2026, 03:28:47) [Clang 19.1.7 ]
** Python executable: /Users/adreitz/pinokio/api/ComfyUI/app/env/bin/python
** ComfyUI Path: /Users/adreitz/pinokio/api/ComfyUI/app
** ComfyUI Base Folder Path: /Users/adreitz/pinokio/api/ComfyUI/app
** User directory: /Users/adreitz/pinokio/api/ComfyUI/app/user
** ComfyUI-Manager config path: /Users/adreitz/pinokio/api/ComfyUI/app/user/__manager/config.ini
** Log path: /Users/adreitz/pinokio/api/ComfyUI/app/user/comfyui.log
[INFO]
Prestartup times for custom nodes:
[INFO] 1.0 seconds: /Users/adreitz/pinokio/api/ComfyUI/app/custom_nodes/ComfyUI-Manager
[INFO]
/Users/adreitz/pinokio/api/ComfyUI/app/env/lib/python3.10/site-packages/torch/cuda/__init__.py:66: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
[INFO] Found comfy_kitchen backend eager: {'available': True, 'disabled': False, 'unavailable_reason': None, 'capabilities': ['adaln', 'apply_rope', 'apply_rope1', 'apply_rope_split_half', 'apply_rope_split_half1', 'convrot_w4a4_linear', 'dequantize_convrot_w4a4_weight', 'dequantize_int8_convrot_weight', 'dequantize_int8_convrot_weight_dtype', 'dequantize_int8_simple', 'dequantize_int8_simple_dtype', 'dequantize_mxfp8', 'dequantize_nvfp4', 'dequantize_per_tensor_fp8', 'gemv_awq_w4a16', 'int8_linear', 'prepare_int4_weight_for_int8_linear', 'quantize_and_rotate_rowwise', 'quantize_convrot_w4a4_weight', 'quantize_int8_convrot_weight', 'quantize_int8_rowwise', 'quantize_int8_tensorwise', 'quantize_mxfp8', 'quantize_nvfp4', 'quantize_per_tensor_fp8', 'quantize_svdquant_w4a4', 'rms_rope', 'rms_rope1', 'rms_rope_split_half', 'rms_rope_split_half1', 'scaled_mm_mxfp8', 'scaled_mm_nvfp4', 'scaled_mm_svdquant_w4a4', 'stochastic_rounding_fp8']}
[INFO] Found comfy_kitchen backend cuda: {'available': False, 'disabled': True, 'unavailable_reason': 'Extension file not found: /Users/adreitz/pinokio/api/ComfyUI/app/env/lib/python3.10/site-packages/comfy_kitchen/backends/cuda/_C.abi3.so', 'capabilities': []}
[INFO] Found comfy_kitchen backend triton: {'available': False, 'disabled': True, 'unavailable_reason': "ImportError: No module named 'triton'", 'capabilities': []}
[INFO] Checkpoint files will always be loaded safely.
[INFO] Total VRAM 65536 MB, total RAM 65536 MB
[INFO] pytorch version: 2.14.0.dev20260706
[INFO] Mac Version (26, 5, 2)
[INFO] Set vram state to: SHARED
[INFO] Device: mps
[INFO] Using sub quadratic optimization for attention, if you have memory or speed issues try using: --use-split-cross-attention
[INFO] Python version: 3.10.20 | packaged by conda-forge | (main, Jun 11 2026, 03:28:47) [Clang 19.1.7 ]
[INFO] ComfyUI version: 0.28.0
[INFO] comfy-aimdo version: 0.4.10
[INFO] comfy-kitchen version: 0.2.22
[INFO] comfyui-frontend-package version: 1.47.10
[INFO] comfyui-workflow-templates version: 0.11.15
[INFO] comfyui-embedded-docs version: 0.5.8
[INFO] comfy-kitchen version: 0.2.22
[INFO] comfy-aimdo version: 0.4.10
[INFO] [Prompt Server] web root: /Users/adreitz/pinokio/api/ComfyUI/app/env/lib/python3.10/site-packages/comfyui_frontend_package/static
[INFO] Asset seeder disabled
[INFO] No OpenGL_accelerate module loaded: Acceleration disabled
[INFO] ### Loading: ComfyUI-Manager (V3.41)
[INFO] [ComfyUI-Manager] network_mode: public
[INFO] [ComfyUI-Manager] ComfyUI per-queue preview override detected (PR #11261). Manager's preview method feature is disabled. Use ComfyUI's --preview-method CLI option or 'Settings > Execution > Live preview method'.
[INFO] ### ComfyUI Version: v0.28.0-33-gfeca51a8 | Released on '2026-07-23'
[INFO] [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/alter-list.json
[INFO] [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/model-list.json
[INFO] [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/github-stats.json
[INFO] [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/custom-node-list.json
[INFO] [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/extension-node-map.json
📊 Initial MPS memory: 54.13GB free / 64.00GB total
[INFO]
Import times for custom nodes:
[INFO] 0.0 seconds: /Users/adreitz/pinokio/api/ComfyUI/app/custom_nodes/websocket_image_save.py
[INFO] 0.0 seconds: /Users/adreitz/pinokio/api/ComfyUI/app/custom_nodes/ComfyUI-ODE
[INFO] 0.0 seconds: /Users/adreitz/pinokio/api/ComfyUI/app/custom_nodes/nightmare_promptgen
[INFO] 0.1 seconds: /Users/adreitz/pinokio/api/ComfyUI/app/custom_nodes/ComfyUI-Manager
[INFO] 1.4 seconds: /Users/adreitz/pinokio/api/ComfyUI/app/custom_nodes/seedvr2_videoupscaler
[INFO]
[INFO] Context impl SQLiteImpl.
[INFO] Will assume non-transactional DDL.
[INFO] Disabling intermediate node cache.
[INFO] Starting server

[INFO] To see the GUI go to: http://127.0.0.1:8188
FETCH ComfyRegistry Data: 5/164
FETCH ComfyRegistry Data: 10/164
[WARNING] [DEPRECATION WARNING] Detected import of deprecated legacy API: /scripts/ui.js. This is likely caused by a custom node extension using outdated APIs. Please update your extensions or contact the extension author for an updated version.
[WARNING] [DEPRECATION WARNING] Detected import of deprecated legacy API: /extensions/core/groupNode.js. This is likely caused by a custom node extension using outdated APIs. Please update your extensions or contact the extension author for an updated version.
[WARNING] [DEPRECATION WARNING] Detected import of deprecated legacy API: /scripts/ui/components/buttonGroup.js. This is likely caused by a custom node extension using outdated APIs. Please update your extensions or contact the extension author for an updated version.
[WARNING] [DEPRECATION WARNING] Detected import of deprecated legacy API: /scripts/ui/components/button.js. This is likely caused by a custom node extension using outdated APIs. Please update your extensions or contact the extension author for an updated version.
FETCH ComfyRegistry Data: 15/164
[INFO] got prompt
[INFO] Using split attention in VAE
[INFO] Using split attention in VAE
FETCH ComfyRegistry Data: 20/164
[INFO] VAE load device: mps, offload device: cpu, dtype: torch.float16
[INFO] Requested to load VideoAutoencoderKLWrapper
[INFO] loaded completely; 478.07 MB loaded, full load: True
FETCH ComfyRegistry Data: 25/164
FETCH ComfyRegistry Data: 30/164
FETCH ComfyRegistry Data: 35/164
FETCH ComfyRegistry Data: 40/164
FETCH ComfyRegistry Data: 45/164
FETCH ComfyRegistry Data: 50/164
FETCH ComfyRegistry Data: 55/164
FETCH ComfyRegistry Data: 60/164
FETCH ComfyRegistry Data: 65/164
FETCH ComfyRegistry Data: 70/164
FETCH ComfyRegistry Data: 75/164
FETCH ComfyRegistry Data: 80/164
FETCH ComfyRegistry Data: 85/164
FETCH ComfyRegistry Data: 90/164
FETCH ComfyRegistry Data: 95/164
[INFO] model weight dtype torch.float16, manual cast: torch.bfloat16
[INFO] model_type FLOW
[INFO] Requested to load SeedVR2
[INFO] 0 models unloaded.
FETCH ComfyRegistry Data: 100/164
[INFO] loaded completely; 6468.73 MB loaded, full load: True
0%| | 0/1 [00:00
run_tile(tile_index, tile_range)
File "/Users/adreitz/pinokio/api/ComfyUI/app/comfy/ldm/seedvr/vae.py", line 155, in run_tile
tile_out = run_temporal_chunks(tile_x)
File "/Users/adreitz/pinokio/api/ComfyUI/app/comfy/ldm/seedvr/vae.py", line 120, in run_temporal_chunks
out = model.decode_(t_chunk)
File "/Users/adreitz/pinokio/api/ComfyUI/app/comfy/ldm/seedvr/vae.py", line 1360, in decode_
decoded = self.slicing_decode(z)
File "/Users/adreitz/pinokio/api/ComfyUI/app/comfy/ldm/seedvr/vae.py", line 1427, in slicing_decode
return self._decode(z)
File "/Users/adreitz/pinokio/api/ComfyUI/app/comfy/ldm/seedvr/vae.py", line 1378, in _decode
output = self.decoder(_z, memory_state=memory_state, memory_cache=memory_cache)
File "/Users/adreitz/pinokio/api/ComfyUI/app/env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/Users/adreitz/pinokio/api/ComfyUI/app/env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
return forward_call(*args, **kwargs)
File "/Users/adreitz/pinokio/api/ComfyUI/app/comfy/ldm/seedvr/vae.py", line 1294, in forward
sample = up_block(sample, latent_embeds, memory_state=memory_state, memory_cache=memory_cache)
File "/Users/adreitz/pinokio/api/ComfyUI/app/env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/Users/adreitz/pinokio/api/ComfyUI/app/env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
return forward_call(*args, **kwargs)
File "/Users/adreitz/pinokio/api/ComfyUI/app/comfy/ldm/seedvr/vae.py", line 1007, in forward
hidden_states = upsampler(hidden_states, memory_state=memory_state, memory_cache=memory_cache)
File "/Users/adreitz/pinokio/api/ComfyUI/app/env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/Users/adreitz/pinokio/api/ComfyUI/app/env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
return forward_call(*args, **kwargs)
File "/Users/adreitz/pinokio/api/ComfyUI/app/comfy/ldm/seedvr/vae.py", line 714, in forward
hidden_states = self.upscale_conv(hidden_states)
File "/Users/adreitz/pinokio/api/ComfyUI/app/env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/Users/adreitz/pinokio/api/ComfyUI/app/env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
return forward_call(*args, **kwargs)
File "/Users/adreitz/pinokio/api/ComfyUI/app/comfy/ops.py", line 565, in forward
return self.forward_comfy_cast_weights(*args, **kwargs)
File "/Users/adreitz/pinokio/api/ComfyUI/app/comfy/ops.py", line 558, in forward_comfy_cast_weights
x = self._conv_forward(input, weight, bias, autopad=autopad)
File "/Users/adreitz/pinokio/api/ComfyUI/app/comfy/ops.py", line 554, in _conv_forward
return super()._conv_forward(input, weight, bias, *args, **kwargs)
File "/Users/adreitz/pinokio/api/ComfyUI/app/env/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 730, in _conv_forward
return F.conv3d(
RuntimeError: MPS backend out of memory (MPS allocated: 3.02 GiB, other allocations: 85.02 GiB, max allowed: 88.13 GiB). Tried to allocate 128.00 MiB on shared pool. Use PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0 to disable upper limit for memory allocations (may cause system failure).

[INFO] Prompt executed in 161.37 seconds
```

### Other

Machine: Macbook Pro M3 Max, 64 GB unified memory, MacOS 26.5.2, PyTorch nightly 2.14.0.dev20260706

Contributor guide

Open the contributing guide

Research direction

Reproduce the attached workflow on macOS with MPS, then inspect nodes.py at the decode entry point and the VAE decode_tiled path; the traceback also identifies execution.py around node execution. Compare memory behavior during tiled encode and decode, and consider the seedvr2_videoupscaler custom node mentioned in the logs. Done means the workflow produces valid output without the reported MPS out-of-memory failure or excessive memory growth.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.