antirez / antirez/h3.c

256×256 halved spatial RoPE swaps prompt subjects on the optimized INT8 package (fixed by --use-reference-rope)

Abierto
#28 0 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
C
Estrellas
2.7k
Forks
210
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

## Summary

The automatic `spatial_rope_scale = 0.5` applied at exactly 256×256 (h3.c,
`h3_generate`, unless `--use-reference-rope`) breaks subject adherence when
used with the Comfy-Org prequantized INT8 package. Prompts render as
coherent but unrelated scenes: the sentence structure and setting class
survive while the subject noun is replaced. Restoring reference RoPE
coordinates fixes it with no other change. The released-BF16 validation of
the halved grid documented in the README does not appear to transfer to
this package.

## Environment

- h3.c commit `8974cc055ea9c02fcd14cc27dfda3e1027c05153`, built with the
stock Makefile (clang, -O3)
- MacBook Pro, Apple M1 Pro, 32 GB unified memory, macOS (Darwin 25.6)
- Model package: `Comfy-Org/MiniMax-H3`, revision
`014cd40f7e177756c6b2473c0d93b1c89a790dd2` (optimized INT8 single-file
layout: `minimax_h3_fl2va_pruned_int8_convrot.safetensors`,
`qwen3vl_32b_minimax_h3_int8_convrot.safetensors`, fp16 video VAE, fp32
audio VAE; SHA-256 verified against the Hugging Face repo)

## Reproduction

All runs: `--width 256 --height 256 --frames 22 --steps 4 --layers 50
--reuse 1`, default seed 42, prompt-only FL2VA.

| # | Prompt | Extra flag | Middle-frame content |
|---|---|---|---|
| 1 | "A red fox walks through fresh snow in a pine forest. Medium tracking shot, natural winter light, realistic fur." | — | A fox (subject OK; no snow/forest) |
| 2 | "A fluffy white cat sits on a sofa in a living room." | — | **A man sitting at a table indoors** |
| 3 | "fluffy cat dance cha-cha-cha" | — | **A herd of livestock in a savanna** |
| 4 | "A red sports car driving fast down a desert highway at sunset, dust trailing behind." | — | **A person standing in a grassy field** |
| 5 | "A fluffy white cat sits on a sofa in a living room." | `--use-reference-rope` | **A cat** (sitting, correct subject) |

Runs 2 and 5 differ only by `--use-reference-rope`. At 512×512 (where no
RoPE halving applies) the same cat prompt adheres correctly at 7 steps.
Frames/MP4s available on request.

Ruled out while isolating this: tokenizer (exact encode/decode round trips,
correct Qwen IDs), step budget (a 15-step 256² run still swapped the
subject), package integrity (pinned hashes), and the int8 DiT block math
(`h3_real_optimized_int8_test` passes with its CPU-reference checks).

## Suggested fix

Gate the automatic 256² half-scale RoPE on the model layout — apply it for
`H3_MODEL_LAYOUT_RELEASED_DIRECTORY` (where it was validated) and default
to reference coordinates for `H3_MODEL_LAYOUT_OPTIMIZED_INT8_SINGLE_FILE`,
or re-validate the halving against the prequantized package. The
`prequantized_int8` flag is already known at DiT load time.

Downstream we work around this by forcing `use_reference_rope` whenever the optimized INT8 layout is loaded; gating the halving on `H3_MODEL_LAYOUT_OPTIMIZED_INT8_SINGLE_FILE` in h3.c itself would fix it for every consumer.

## Related: the optimized Qwen test verifies no numerics

While isolating this we noticed `tests/test_real_optimized_qwen.c` only
checks that outputs are finite, non-zero, and memory-bounded for token IDs
{0, 1} — unlike `test_real_optimized_int8.c`, which checks projections
against CPU references. A numeric parity check for the optimized text
encoder would have shortened this investigation considerably and would
catch regressions in the ConvRot/int8 language path.

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.