mudler / mudler/vllm.cpp

LTX-2.5 VAE: upstream's bf16 noise draw is a DIFFERENT sequence, and mirroring it invalidates every captured render digest

Open
#2,780 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: -

Owed by .agents/specs/ltx25-a24-video-vae-bf16.md (A24 wave 3), which is
drafted but not yet landed; this issue is filed ahead of it so the measurement is
not lost with the draft. Found while specifying wave 3 of gap A24.

This is a product decision, not a mirror question. It is filed for the
developer, and no row implements it until it is answered.

The measurement

Upstream's video VAE draws its noise from a different sequence at bfloat16
than at float32. Measured by executing Lightricks/LTX-2 @ fd4ded7f
(conv_video_decoder.py:288-294 into resnet.py:115):

  • 1151 of 1152 drawn values differ between the two dtypes.
  • Only 9 of 64 bf16 draws appear anywhere in the first 256 f32 draws.

So it is not a rounding of the f32 stream. It is a different stream.

Why it is a decision

Gap A24 mirrors upstream's single model dtype: distilled.py:109 sets
self.dtype = torch.bfloat16 and hands it to every component. Waves 1 and 2
(the text tower, 8e582a5f9, and the connector, #2753) mirrored it exactly,
because AGENTS.md says to mirror vLLM wherever it defines behaviour and never to
ask how a mirrored feature must behave.

Wave 3 cannot follow that rule silently. Mirroring the bf16 noise draw would
change every captured render digest in this repository — the goldens, the
committed frame digests, and the adherence-campaign renders that
.agents/specs/ltx25-adherence-rescore.md established are bit-deterministic and
whose control arm is byte-identical across 454 commits.

That is a scale of record invalidation a row should not choose on its own.

What a decision needs to answer

  • (a) Mirror the bf16 draw, and re-capture every affected digest in the same
    change, recording the invalidation explicitly.
  • (b) Keep the f32 noise stream as a named, recorded divergence from
    upstream, with the reason and the measurement above, so the digests stay
    comparable across the campaign.
  • (c) Mirror it but gate the VAE arm on a fresh set of digests captured
    after the switch, leaving pre-existing renders anchored to the old stream and
    marked as such.

Unverified

Whether the shipped bench geometries produce more than one temporal group, which
decides whether the tiled-blend path (a separate 31%-of-values effect measured in
the same sweep) is reachable in practice. It needs a real-weights render and a
GPU lease; none was taken.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read .agents/specs/ltx25-a24-video-vae-bf16.md, distilled.py:109, and the upstream conv_video_decoder.py:288-294 and resnet.py:115 measurements. The work is blocked on choosing and recording one of the three noise-stream policies; if mirroring is selected, affected render digests and the adherence-rescore records must be addressed, while the tiled-blend path remains unverified without a real-weights GPU render.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, pytorch
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.