Comfy-Org / Comfy-Org/ComfyUI

MiniMax H3 video VAE: decoding a single latent frame (off the 17k+5 grid) produces heavy artifacts — pad-to-2-tokens works; suggest graceful handling or validation

Open
#15,416 2 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
133k
Forks
15.7k
Avg merge
1d 7h
Merged PRs (30d)
158

Description

## Summary

The MiniMax H3 video VAE **decode** path produces visibly degraded output — 256px tile seams and banding in its default tiled mode, and strong ViT patch-grid artifacts untiled. The **encode** path is fine.

## Repro (master a464ac33, single 1024x1024 image, `minimax_h3_video_vae_fp16.safetensors`)

`LoadImage -> VAEEncode -> VAEDecode -> SaveImage`, compare output to input (mean abs pixel error, 0-255):

| path | roundtrip error |
|---|---|
| reference implementation (see below) | **4.6** |
| ComfyUI decode, tiled (default `tiling=True`, 256/64) | **31.4** — visible tile seams + banding |
| ComfyUI decode, `tiling=False` | **93.7** — 16px patch-grid artifacts |

Errors are dtype-independent (fp16 / `--bf16-vae` / `--fp32-vae` within 0.1 of each other), so it's algorithmic, not precision.

**Encode is healthy**: comfy-encoded latents match the reference encoder at cosine 0.9997, and decoding a comfy-encoded latent through the reference decoder gives the same 4.6 as a full reference roundtrip.

## Reference implementation

The ai-toolkit MiniMax-H3 extension carries an independent port of this VAE ([ostris/ai-toolkit `extensions_built_in/diffusion_models/minimax_h3/src/vae.py`](https://github.com/ostris/ai-toolkit)) that implements the same 256px/64px-overlap spatial windowing with linear blending and reproduces the released pipeline's output (4.6 roundtrip on the same weights file). Diffing the tile layout / per-tile RoPE origin / blend-weight math between the two implementations should localize the defect quickly — the untiled 93.7 result also suggests the ViT decoder is only valid in <=256px windows, so the tiled path is the one worth fixing.

Happy to provide the exact test script/images or help bisect. Found while building a custom node that does per-slot decodes of single-frame H3 latents, where the seams are conspicuous on flat backgrounds; the same degradation applies to every normal H3 video decode at >=512px.

## Environment
ComfyUI master `a464ac33`, Linux, RTX PRO 6000 (Blackwell), torch 2.14.0.dev+cu132, `--gpu-only`.

Contributor guide

Open the contributing guide

Research direction

Start by locating the MiniMax H3 VAE decode path behind the VAEDecode entry point, then compare its tile layout, per-tile RoPE origin, and blend-weight math with extensions_built_in/diffusion_models/minimax_h3/src/vae.py from the referenced ai-toolkit implementation. Reproduce the single-frame 1024x1024 roundtrip and verify that tiled decoding removes seams and banding while preserving the reported reference error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.