mudler / mudler/vllm.cpp

gen-ltx2-vae-goldens.py emits thread-count-dependent goldens, so the oracle only reproduces at one core count

Open
#2,855 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: -

Needs an owner. Found during the fresh review of #2852 (A24 wave 4); it is
pre-existing and was deliberately left out of that row's scope.

The problem

scripts/gen-ltx2-vae-goldens.py emits goldens that depend on the torch
thread count
. Regenerating the committed file at OMP_NUM_THREADS=2 moved
4970 lines of previously committed goldens. At the default thread count
(nproc on this box) the file regenerates byte-identically.

So the oracle is reproducible only on a machine with the same core count as
whoever last generated it. That silently violates what a golden is for.

What is NOT affected

Section 6e, added by #2852, was checked at both thread counts and is
thread-invariant. The f32 sections are the ones that move. #2852's own
regeneration is a pure addition (48 insertions, 0 deletions) at the default
thread count, so nothing it landed depends on this.

Why it matters

.agents/verification.md treats a regenerated golden as evidence that the port
still matches upstream. If the file moves under an unrelated variable, then:

  • a contributor on a different machine sees thousands of spurious diffs and
    cannot tell them from a real divergence;
  • "the goldens regenerate byte-identically" stops being a check anybody can run
    and become a check only one machine can run;
  • a genuine defect landing at the same time as a thread-count change would be
    camouflaged by the noise.

What a fix needs to decide

  • (a) Pin the thread count inside the generator (torch.set_num_threads(1)
    or an explicit value) and regenerate once, accepting a large one-time diff.
  • (b) Find the reductions that are thread-order dependent and make the
    generator compute them deterministically, leaving the values unchanged.
  • (c) Record the required thread count as part of the oracle's identity, so
    a mismatched regeneration is a refusal rather than a diff.

(a) is the smallest and (c) is the most honest; (b) is the only one that makes
the numbers themselves stable. This needs a spec, because it changes what a
golden means here.

Reproduction

cd /home/mudler/_git/vllm.cpp
python3 scripts/gen-ltx2-vae-goldens.py --ltx2 ~/_git/LTX-2   # byte-identical
OMP_NUM_THREADS=2 python3 scripts/gen-ltx2-vae-goldens.py --ltx2 ~/_git/LTX-2
git diff --stat tests/vllm/models/ltx2_vae_goldens.inc         # ~4970 lines

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with scripts/gen-ltx2-vae-goldens.py and reproduce the byte-identical run, then compare it with OMP_NUM_THREADS=2 using tests/vllm/models/ltx2_vae_goldens.inc. Read .agents/verification.md to determine the oracle contract and resolve whether to pin threads, make reductions deterministic, or record the required thread count. Done means the chosen policy is specified and regeneration no longer produces unexplained thread-count-dependent differences.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
testing, tooling
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.