Comfy-Org / Comfy-Org/ComfyUI

MiniMax H3 (local): text prompt has zero effect on output — same seed + different prompts produce pixel-identical videos

Open
#15,805 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
133k
Forks
15.7k
Avg merge
1d 7h
Merged PRs (30d)
158

Description

# GitHub Issue Draft: MiniMax H3 — text conditioning has zero effect on output (same seed + different prompts → bit-identical videos)

## Summary

When running MiniMax H3 locally via the built-in `MiniMaxH3ImageToVideo` node, the text prompt has **no effect whatsoever on the generated output**. Two completely different prompts with the same seed produce **pixel-identical videos**. The model appears to run in an unconditional mode: sampling works (different seeds → different outputs), but conditioning never influences generation. The decoded output is always a structured mosaic of 16×16 colored blocks (green/magenta pattern), which looks like the VAE faithfully decoding latents that have patch-aligned structure but no semantic content.

## Environment

- ComfyUI: v0.33.x (tested on `72865f4f` v0.33.1 and latest master `7dde5617`)
- OS: Windows 11, GPU: NVIDIA RTX 4060 Ti 16GB, torch 2.12.1+cu130
- comfy-aimdo 0.4.13, comfy-kitchen 0.2.31
- Models:
- Diffusion: `minimax_h3_fl2va_pruned_int8_convrot.safetensors` (19.53 GB, from Comfy-Org/MiniMax-H3)
- Text encoder: `qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors` (14.61 GB)
- VAEs: `minimax_h3_video_vae_fp16.safetensors`, `minimax_h3_audio_vae_fp32.safetensors`
- LoRA (optional): `minimax_h3_fl2v_turbo_8step_v1.0_comfyui_bf16.safetensors`
- Submitted via `POST /prompt` (API format). Also reproduced via UI-built graph.

## Minimal reproduction

Standard t2v chain:

```
UNETLoader → LoraLoader(Turbo, optional) → MiniMaxH3SigmaShift(12/3)
CLIPLoader(type=minimax) ──────────────────────────────┐
VAELoader(video) ─────────────────────────────────────┐│
↓↓
MiniMaxH3ImageToVideo(prompt=X, 832x480, length=22)
│ CONDITIONING │ LATENT
↓ ↓
BasicGuider SamplerCustomAdvanced
└────── VAEDecode → CreateVideo → SaveVideo
```

Run A: prompt = "A cute orange tabby cat sitting on a windowsill..." , seed = 111
Run B: prompt = "A vast dark ocean at night with huge waves crashing...", seed = 111

**Result: outputs are pixel-identical (mean abs diff = 0.0000 across all frames).**

Control: same prompt, seed 111 vs seed 999 → outputs differ (mean abs diff ≈ 116), so the sampler/noise path works.

This holds with and without the Turbo LoRA, at length 22 and length 124 (in-training-range), at 864×480 and 832×480.

## What I verified (so the report narrows the search space)

I instrumented the graph with a passthrough diagnostic node and checked every stage:

1. **Server receives correct inputs** — `/history` shows both graphs carried their distinct `prompt` strings into node 6 (`MiniMaxH3ImageToVideo`). Not an API-format field-dropping issue.
2. **Text encoder works** — loading the TE directly via `comfy.sd.load_clip(CLIPType.MINIMAX)` outside the server, "cat…" encodes to `[1, 26, 5120]`, "ocean…" to `[1, 23, 5120]`. Different token counts, different embeddings. Tokenizer fine.
3. **Conditioning content is correct inside the graph** — a passthrough node logged, for each prompt:
```
emb shape=[1, 26, 5120] mean=0.153 std=42.85
meta keys: ['minimax_token_tags', 'pooled_output']
minimax_token_tags values: {1: 26} ← all text-modality tags, correct
```
So `clip.tokenize` → `encode_from_tokens_scheduled` produces sane, distinct conditionings that reach the guider.
4. **VAE is fine** — decoding the *empty* AV latent from `MiniMaxH3ImageToVideo` (skipping sampling entirely) yields a uniform dark-gray image, as expected for zeros. Decode path healthy.
5. **Not the comfy-kitchen CUDA INT8 kernels** — force-disabled the CUDA backend (`ck.registry.disable("cuda")`, forcing eager/emulated dequant) and re-ran: same-prompt output differs from the CUDA-backend run by only 0.0096 (float noise), different-prompt still 0.0000. Kernels exonerated.
6. **Not aimdo/DynamicVRAM paging corruption** — identical blocky output with DynamicVRAM enabled and with `--disable-dynamic-vram` (legacy lowvram loader).
7. **Not the LoRA** — reproduces with `strength_clip=0`, `strength_model=0`, and with LoRA node removed entirely.
8. **Not guider choice** — `BasicGuider` and `CFGGuider(cfg=4, negative=)` produce identical results.
9. **No runtime errors** — runs report success; no NaN warnings; models load fully ("208 patches attached" with Turbo).

## Interpretation

Everything upstream of the DiT is correct and distinct per prompt, yet the sampled latent does not depend on the prompt at all and has deterministic patch-aligned block structure. This points at the conditioning → DiT handoff inside the H3 forward path: e.g. the packed sequence built in `MiniMaxH3.extra_conds` / `PackedLayout` / `_forward` not actually admitting the `c_crossattn` rows into the attention computation on this setup, leaving the model to denoise unconditionally.

Suspect area: interaction between the `int8_convrot` weight format of `minimax_h3_fl2va_pruned_int8_convrot.safetensors` and the H3 packed-layout forward in ComfyUI 0.33.x (possibly the `preprocess_text_embeds` / `condition_proj` path, or cross-attn rows being dropped when building the packed layout).

## Requested

- Confirmation whether others can reproduce with `minimax_h3_fl2va_pruned_int8_convrot` on NVIDIA (esp. RTX 40-series), API or UI submission.
- Any known-good combo (ComfyUI version / backend flags / alternative quantized checkpoint) where text conditioning demonstrably changes the output with this checkpoint.

Happy to provide full workflow JSONs, logs, and output frames.

Contributor guide

Open the contributing guide

Research direction

Trace the H3 conditioning handoff through MiniMaxH3.extra_conds, PackedLayout, and _forward, focusing on preprocess_text_embeds and condition_proj. Compare the packed inputs and attention path for the two prompts, then reproduce with the listed checkpoint and workflow. Done means identical-seed runs with distinct prompts produce different sampled latents or decoded videos while the existing seed control remains valid.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
backend, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.