antirez / antirez/h3.c

256×256 halved spatial RoPE swaps prompt subjects on the optimized INT8 package (fixed by --use-reference-rope)

未关闭
#28 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
C
星标
2.7k
派生
210
PR 合并指标
30 天内没有已合并 PR

描述

## Summary

The automatic `spatial_rope_scale = 0.5` applied at exactly 256×256 (h3.c,
`h3_generate`, unless `--use-reference-rope`) breaks subject adherence when
used with the Comfy-Org prequantized INT8 package. Prompts render as
coherent but unrelated scenes: the sentence structure and setting class
survive while the subject noun is replaced. Restoring reference RoPE
coordinates fixes it with no other change. The released-BF16 validation of
the halved grid documented in the README does not appear to transfer to
this package.

## Environment

- h3.c commit `8974cc055ea9c02fcd14cc27dfda3e1027c05153`, built with the
stock Makefile (clang, -O3)
- MacBook Pro, Apple M1 Pro, 32 GB unified memory, macOS (Darwin 25.6)
- Model package: `Comfy-Org/MiniMax-H3`, revision
`014cd40f7e177756c6b2473c0d93b1c89a790dd2` (optimized INT8 single-file
layout: `minimax_h3_fl2va_pruned_int8_convrot.safetensors`,
`qwen3vl_32b_minimax_h3_int8_convrot.safetensors`, fp16 video VAE, fp32
audio VAE; SHA-256 verified against the Hugging Face repo)

## Reproduction

All runs: `--width 256 --height 256 --frames 22 --steps 4 --layers 50
--reuse 1`, default seed 42, prompt-only FL2VA.

| # | Prompt | Extra flag | Middle-frame content |
|---|---|---|---|
| 1 | "A red fox walks through fresh snow in a pine forest. Medium tracking shot, natural winter light, realistic fur." | — | A fox (subject OK; no snow/forest) |
| 2 | "A fluffy white cat sits on a sofa in a living room." | — | **A man sitting at a table indoors** |
| 3 | "fluffy cat dance cha-cha-cha" | — | **A herd of livestock in a savanna** |
| 4 | "A red sports car driving fast down a desert highway at sunset, dust trailing behind." | — | **A person standing in a grassy field** |
| 5 | "A fluffy white cat sits on a sofa in a living room." | `--use-reference-rope` | **A cat** (sitting, correct subject) |

Runs 2 and 5 differ only by `--use-reference-rope`. At 512×512 (where no
RoPE halving applies) the same cat prompt adheres correctly at 7 steps.
Frames/MP4s available on request.

Ruled out while isolating this: tokenizer (exact encode/decode round trips,
correct Qwen IDs), step budget (a 15-step 256² run still swapped the
subject), package integrity (pinned hashes), and the int8 DiT block math
(`h3_real_optimized_int8_test` passes with its CPU-reference checks).

## Suggested fix

Gate the automatic 256² half-scale RoPE on the model layout — apply it for
`H3_MODEL_LAYOUT_RELEASED_DIRECTORY` (where it was validated) and default
to reference coordinates for `H3_MODEL_LAYOUT_OPTIMIZED_INT8_SINGLE_FILE`,
or re-validate the halving against the prequantized package. The
`prequantized_int8` flag is already known at DiT load time.

Downstream we work around this by forcing `use_reference_rope` whenever the optimized INT8 layout is loaded; gating the halving on `H3_MODEL_LAYOUT_OPTIMIZED_INT8_SINGLE_FILE` in h3.c itself would fix it for every consumer.

## Related: the optimized Qwen test verifies no numerics

While isolating this we noticed `tests/test_real_optimized_qwen.c` only
checks that outputs are finite, non-zero, and memory-bounded for token IDs
{0, 1} — unlike `test_real_optimized_int8.c`, which checks projections
against CPU references. A numeric parity check for the optimized text
encoder would have shortened this investigation considerably and would
catch regressions in the ConvRot/int8 language path.

贡献指南

这个仓库没有索引到贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。