MiniMax H3 i2v crashes in patchify_video when (width mod 32) >= 16
- Dominant language
- Python
- Stars
- 133k
- Forks
- 15.7k
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 153
Description
### Expected Behavior
`MiniMaxH3ImageToVideo` with a `first_frame` should sample successfully for any width/height the node accepts, the same way it already does when no keyframe is supplied.
### Actual Behavior
Sampling crashes inside the DiT forward pass, **after** the text encoder, the VAE encode and the model load have all completed:
```
RuntimeError: shape '[1, 24, 1, 1, 20, 2, 15, 2]' is invalid for input of size 29760
File "comfy/ldm/minimax/model.py", line 648, in _forward
cond_video_rows = self._cond_video_rows(payload, device)
File "comfy/ldm/minimax/model.py", line 506, in _cond_video_rows
r = patchify_video(z.to(torch.float32), self.patch_size)
File "comfy/ldm/minimax/model.py", line 47, in patchify_video
x = latent.reshape(b, c, t, pt, h, ph, w, pw)
```
Reported node is `SamplerCustomAdvanced`. It is fully deterministic — the same width always fails.
We see this ~31 times per 12h across our fleet on user-authored workflows.
### Steps to Reproduce
Minimal graph: `UNETLoader` → `MiniMaxH3ImageToVideo` (**width=496**, height=640, length=5, any `first_frame`) → `BasicGuider` → `SamplerCustomAdvanced` → `VAEDecode` → `SaveImage`.
- `width=496` → crash
- `width=512` → fine
- `width=520` → fine
- t2v (no `first_frame`/`last_frame`) → fine at any width
### Root cause
`patchify_video` (`comfy/ldm/minimax/model.py:42`) floor-divides the grid and then reshapes as if the division were exact:
```python
t, h, w = t_full // pt, h_full // ph, w_full // pw # floor
x = latent.reshape(b, c, t, pt, h, ph, w, pw) # requires exact divisibility
```
The **main** latent never hits this, because `_forward` pads it and crops the result back:
```python
:555 orig_t, orig_h, orig_w = video_x.shape[2], video_x.shape[3], video_x.shape[4]
:557 video_x = comfy.ldm.common_dit.pad_to_patch_size(video_x, self.patch_size)
...
:729 video_out = video_out[:, :, :orig_t, :orig_h, :orig_w]
```
**`_cond_video_rows` (`:504-506`) has no equivalent pad**, so keyframe and ref latents are patchified raw. That is why every observed failure has `t = 1` — it is always a single-frame condition latent, never the main video latent (patchified separately at `:646`).
This is also a latent *correctness* issue independent of the crash: `PackedLayout` sizes each keyframe's `cond` segment as `vt * frame_rows`, where `frame_rows` comes from `_frame_grid` on the **padded** target grid (keyframes explicitly "share the target spatial grid", `:345`). An unpadded keyframe can never match that allocation.
### Trigger condition
Not "width is not a multiple of 32" — `vae_encode_crop_pixels` (`comfy/sd.py:1167`) first crops each pixel dim down to a multiple of `spacial_compression_encode()` = 16, so the condition latent is `floor(px / 16)` and only an **odd** result breaks. Verified exhaustively over px ∈ [32, 4096], no counter-examples:
```
crash <=> (px mod 32) >= 16 on width or height
```
| px mod 32 | VAE crops to | latent | result |
|---|---|---|---|
| 0–15 (512, 520, 1350) | 512 / 512 / 1344 | 32 / 32 / 84 — even | fine |
| 16–31 (496, 500, 528) | 496 / 496 / 528 | 31 / 31 / 33 — odd | **crash** |
So half the non-multiples of 32 were never affected, which makes this easy to miss.
### Workaround we are running — and why it is not a proper fix
One line in `_cond_video_rows`, giving the condition latents the same treatment the main latent already gets:
```python
for z in payload.get("cond_video_latents", []):
+ z = comfy.ldm.common_dit.pad_to_patch_size(z, self.patch_size)
r = patchify_video(z.to(torch.float32), self.patch_size)
```
Verified end-to-end: `width=496` reproduces the error above before the change and returns a correct **496×640** result after it. Output resolution is unchanged, because the pad lives only in the condition rows, which are never decoded. On already-aligned input the pad is bit-identical to a no-op.
**But padding a condition latent means inventing content that does not exist**, and it is measurable. With a deliberately extreme probe (left half red / right half blue, so the frame's leftmost and rightmost columns are maximally different), the default `circular` mode wraps the red left edge onto the new rightmost column and it bleeds ~12 px into the result:
| padding_mode | redness delta at the right edge (probe) | same metric on a real photo |
|---|---:|---:|
| no pad (aligned width, baseline) | +2.3 | 23.9 |
| `circular` (current default) | **+108.1** | **24.6** |
| `replicate` | +288.2 | 58.7 |
On an ordinary photograph `circular` is indistinguishable from the no-pad baseline (24.6 vs 23.9, identical per-column profile), so in practice this is mild. Two notes that may be useful:
- **`replicate` is consistently worse**, 2.4–2.7× on both inputs. We tried it expecting the opposite. Speculation: the main latent has always been `circular`-padded at `:557`, so that is the only boundary pattern the model has seen.
- The bleed reaches visible pixels through **attention** over the padded condition token, not through decode — the padded column is dropped at `:729` before `VAEDecode`.
### What would be a proper fix
Any fill value is a guess about content that isn't there. The clean solution is probably to **exclude the padded condition tokens from attention** (mask them in `PackedLayout` / the attention mask) rather than to pick a nicer fill — or alternatively to handle the odd-grid case in `patchify_video`/`PackedLayout` consistently so no invented tokens are needed at all. Both are bigger changes than we felt comfortable making blind, so we are reporting rather than sending a PR.
Happy to test any patch against the reproduction above.
### Environment
- ComfyUI `0.33.0`, and the code is unchanged on `master` as of `76135e557`
- MiniMax H3, `minimax_h3_fl2va_pruned_int8_convrot` + `minimax_h3_video_vae_fp16`
- RTX 5090, torch 2.x, `--use-sage-attention --fast fp8_matrix_mult`
- Runs are bit-identical across process restarts, so the measurements above are reproducible
Contributor guide
Research direction
Start with comfy/ldm/minimax/model.py, especially patchify_video, _cond_video_rows, and the PackedLayout sizing described in the report; reproduce the 496×640 first-frame workflow. Trace how condition rows and padded target grids interact, then verify that the crash is gone and conditioning does not introduce invalid or invented attention tokens.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- ai, machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100