Comfy-Org / Comfy-Org/ComfyUI

OOM when Minimax H3 VAE encoding a video

Open
#15,312 11 comments 0 reactions 0 assignees View on GitHub
Potential Bug
Dominant language
Python
Stars
133k
Forks
15.7k
Avg merge
1d 7h
Merged PRs (30d)
158

Description

### Custom Node Testing

- [x] I have tried disabling custom nodes and the issue persists (see [how to disable custom nodes](https://docs.comfy.org/troubleshooting/custom-node-issues#step-1%3A-test-with-all-custom-nodes-disabled) if you need help)

### Expected Behavior

I'm currently using a AMD rx9070 XT gpu, which has 16 GB Vram. I was confronted with OOM when using the r2v workflow with a video input.

### Actual Behavior

OOM because of im2col when computing in Conv. Actually I don't know how to solve it, but I ask claude code to make it computed by fp8 and it finally run successfully.

### Steps to Reproduce

Minimax h3, r2v workflow with a video input.

### Debug Logs

```powershell
[2026-08-05 13:30:36.698] Requested to load MiniMaxH3VideoVAE
[2026-08-05 13:30:37.266] Model MiniMaxH3VideoVAE prepared for dynamic VRAM loading. 4965MB Staged. 0 patches attached. Force pre-loaded 128 weights: 348 KB.
[2026-08-05 13:30:50.977] Model MiniMaxH3VideoVAE prepared for dynamic VRAM loading. 4965MB Staged. 0 patches attached. Force pre-loaded 128 weights: 348 KB.
[2026-08-05 13:30:52.804] Warning: Ran out of memory when regular VAE encoding, retrying with tiled VAE encoding.
[2026-08-05 13:30:54.349] !!! Exception during processing !!! CUDA out of memory. Tried to allocate 7.17 GiB. GPU 0 has a total capacity of 15.92 GiB of which 2.50 GiB is free. Of the allocated memory 6.19 GiB is allocated by PyTorch, and 6.88 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.
[2026-08-05 13:30:54.441] Traceback (most recent call last):
File "/home/user/github/ComfyUI/execution.py", line 545, in execute
output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(...)
File "/home/user/github/ComfyUI/execution.py", line 344, in get_output_data
return_values = await _async_map_node_over_list(...)
File "/home/user/github/ComfyUI/execution.py", line 318, in _async_map_node_over_list
await process_inputs(input_dict, i)
File "/home/user/github/ComfyUI/execution.py", line 306, in process_inputs
result = f(**inputs)
File "/home/user/github/ComfyUI/comfy_api/internal/__init__.py", line 149, in wrapped_func
return method(locked_class, **inputs)
File "/home/user/github/ComfyUI/comfy_api/latest/_io.py", line 1935, in EXECUTE_NORMALIZED
to_return = cls.execute(*args, **kwargs)
File "/home/user/github/ComfyUI/comfy_extras/nodes_minimax_h3.py", line 254, in execute
z = vae.encode(frames)
File "/home/user/github/ComfyUI/comfy/sd.py", line 1297, in encode
samples = self._encode_tiled_owned(pixel_samples, tile_x=tile, tile_y=tile, overlap=overlap)
File "/home/user/github/ComfyUI/comfy/sd.py", line 1138, in _encode_tiled_owned
out = self.first_stage_model.encode_tiled(x, **kwargs)
File "/home/user/github/ComfyUI/comfy/ldm/minimax/vae.py", line 675, in encode_tiled
return self.encode(x)
File "/home/user/github/ComfyUI/comfy/ldm/minimax/vae.py", line 665, in encode
moments = self.encode_temporal(x)
File "/home/user/github/ComfyUI/comfy/ldm/minimax/vae.py", line 533, in encode_temporal
z_list.append(self._adaptive_encode(clip_x))
File "/home/user/github/ComfyUI/comfy/ldm/minimax/vae.py", line 392, in _adaptive_encode
return self.tiled_encode(x)
File "/home/user/github/ComfyUI/comfy/ldm/minimax/vae.py", line 461, in tiled_encode
row.append(self._encode_moments(tile))
File "/home/user/github/ComfyUI/comfy/ldm/minimax/vae.py", line 385, in _encode_moments
return self.quant_conv(self.encoder(x))
...
File "/home/user/github/ComfyUI/comfy/ldm/minimax/vae.py", line 159, in forward
h = self.down[i_level].block[i_block](h)
...
File "/home/user/github/ComfyUI/comfy/ldm/minimax/vae.py", line 106, in forward
h = self.conv1(F.silu(self.norm1(x), inplace=True))
...
File "/home/user/github/ComfyUI/comfy/ldm/minimax/vae.py", line 55, in forward
return super().forward(x)
File "/home/user/github/ComfyUI/comfy/ops.py", line 609, in forward
return self.forward_comfy_cast_weights(*args, **kwargs)
File "/home/user/github/ComfyUI/comfy/ops.py", line 602, in forward_comfy_cast_weights
x = self._conv_forward(input, weight, bias, autopad=autopad)
File "/home/user/github/ComfyUI/comfy/ops.py", line 598, in _conv_forward
return super()._conv_forward(input, weight, bias, *args, **kwargs)
File ".../torch/nn/modules/conv.py", line 730, in _conv_forward
return F.conv3d(
input, weight, bias, self.stride, self.padding, self.dilation, self.groups
)
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 7.17 GiB. GPU 0 has a total capacity of 15.92 GiB of which 2.50 GiB is free. Of the allocated memory 6.19 GiB is allocated by PyTorch, and 6.88 GiB is reserved by PyTorch but unallocated.
```

### Other

_No response_

Contributor guide

Open the contributing guide

Research direction

Reproduce the Minimax H3 r2v workflow with video input on the reported 16 GB GPU, then trace the failure from comfy_extras/nodes_minimax_h3.py through comfy/ldm/minimax/vae.py and the convolution path in comfy/ops.py. Use the supplied traceback to identify the memory-heavy encoding step; done means the workflow completes without the reported out-of-memory failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.