antirez / antirez/h3.c

256×256 halved spatial RoPE swaps prompt subjects on the optimized INT8 package (fixed by --use-reference-rope)

Offen
#28 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
C
Sterne
2.7k
Forks
210
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

## Summary

The automatic `spatial_rope_scale = 0.5` applied at exactly 256×256 (h3.c,
`h3_generate`, unless `--use-reference-rope`) breaks subject adherence when
used with the Comfy-Org prequantized INT8 package. Prompts render as
coherent but unrelated scenes: the sentence structure and setting class
survive while the subject noun is replaced. Restoring reference RoPE
coordinates fixes it with no other change. The released-BF16 validation of
the halved grid documented in the README does not appear to transfer to
this package.

## Environment

- h3.c commit `8974cc055ea9c02fcd14cc27dfda3e1027c05153`, built with the
stock Makefile (clang, -O3)
- MacBook Pro, Apple M1 Pro, 32 GB unified memory, macOS (Darwin 25.6)
- Model package: `Comfy-Org/MiniMax-H3`, revision
`014cd40f7e177756c6b2473c0d93b1c89a790dd2` (optimized INT8 single-file
layout: `minimax_h3_fl2va_pruned_int8_convrot.safetensors`,
`qwen3vl_32b_minimax_h3_int8_convrot.safetensors`, fp16 video VAE, fp32
audio VAE; SHA-256 verified against the Hugging Face repo)

## Reproduction

All runs: `--width 256 --height 256 --frames 22 --steps 4 --layers 50
--reuse 1`, default seed 42, prompt-only FL2VA.

| # | Prompt | Extra flag | Middle-frame content |
|---|---|---|---|
| 1 | "A red fox walks through fresh snow in a pine forest. Medium tracking shot, natural winter light, realistic fur." | — | A fox (subject OK; no snow/forest) |
| 2 | "A fluffy white cat sits on a sofa in a living room." | — | **A man sitting at a table indoors** |
| 3 | "fluffy cat dance cha-cha-cha" | — | **A herd of livestock in a savanna** |
| 4 | "A red sports car driving fast down a desert highway at sunset, dust trailing behind." | — | **A person standing in a grassy field** |
| 5 | "A fluffy white cat sits on a sofa in a living room." | `--use-reference-rope` | **A cat** (sitting, correct subject) |

Runs 2 and 5 differ only by `--use-reference-rope`. At 512×512 (where no
RoPE halving applies) the same cat prompt adheres correctly at 7 steps.
Frames/MP4s available on request.

Ruled out while isolating this: tokenizer (exact encode/decode round trips,
correct Qwen IDs), step budget (a 15-step 256² run still swapped the
subject), package integrity (pinned hashes), and the int8 DiT block math
(`h3_real_optimized_int8_test` passes with its CPU-reference checks).

## Suggested fix

Gate the automatic 256² half-scale RoPE on the model layout — apply it for
`H3_MODEL_LAYOUT_RELEASED_DIRECTORY` (where it was validated) and default
to reference coordinates for `H3_MODEL_LAYOUT_OPTIMIZED_INT8_SINGLE_FILE`,
or re-validate the halving against the prequantized package. The
`prequantized_int8` flag is already known at DiT load time.

Downstream we work around this by forcing `use_reference_rope` whenever the optimized INT8 layout is loaded; gating the halving on `H3_MODEL_LAYOUT_OPTIMIZED_INT8_SINGLE_FILE` in h3.c itself would fix it for every consumer.

## Related: the optimized Qwen test verifies no numerics

While isolating this we noticed `tests/test_real_optimized_qwen.c` only
checks that outputs are finite, non-zero, and memory-bounded for token IDs
{0, 1} — unlike `test_real_optimized_int8.c`, which checks projections
against CPU references. A numeric parity check for the optimized text
encoder would have shortened this investigation considerably and would
catch regressions in the ConvRot/int8 language path.

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.