mudler / mudler/vllm.cpp

The LTX-2.5 text tower computes a [T, 262144] lm_head, a SoftCap and a full download that nothing reads, twice per render

Open
#2,479 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: - (listed under ## Owed in .agents/specs/ltx25-device-arm-survey.md)

Found in flow by LTX25-DEVICE-ARM-SURVEY while establishing the text tower's
device-arm shape.

Gemma4Model::ForwardHiddenStates returns hidden_states and logits, and
ForwardBody (src/vllm/model_executor/models/gemma4.cpp:733-756) computes the
tied lm_head over EVERY row unless logits_indices is non-empty and shorter
than T (:738-739).

src/vllm/model_executor/models/ltx2_text_encoder.cpp:1166 calls it WITHOUT
logits_indices, so do_gather is false. The call site then reads only
run.hidden_states (:1167-1187). The [T, 262144] f32 buffer, the
vt::MatmulBT against the [262144, 3840] tied embedding table, the SoftCap
and the full Download into out.logits are all discarded.

This runs twice per render — the positive prompt at ltx2_video.cpp:2501 and
the negative one at the guiders pass — on the host, on the default
ti2vid_one_stage path.

Sizing (arithmetic, NOT a measurement)

At the shipped tower geometry (tests/vllm/models/ltx2_gemma4_text_config.json:
hidden_size = 3840, vocab_size = 262144, num_hidden_layers = 48,
intermediate_size = 15360) the lm_head is 2 * 3840 * 262144 = 2.01 GFLOP per
token against roughly 2.16e10 FLOP per token for the 48 decoder layers —
about 8.5% of the tower's GEMM work.

Against the 82.336 s of measured tower on the reference render (rc job
93a60151, n = 3, conditioning.tower 27.014 + guiders.tower 55.322, wall
302.954 s) that is of order 6 to 7 s, ~2.2% of the wall, plus a
T * 262144 * 4-byte allocation and download per pass.

No measurement was taken. This is arithmetic against a measured leaf and must
not be quoted as a measured saving.

The change, and why it is not made yet

Passing logits_indices = {0} at the one call site removes it, and it cannot
change the conditioning: capture(dnorm) (gemma4.cpp:731) records the final
hidden state BEFORE the lm_head.

The gate is the open question. A test can hold the seam
(out.logits.size() == vocab rather than T * vocab), but holding the LTX call
site itself needs an instrument this tree does not expose, and a mutation that
moves nothing is not evidence of anything. That gate design is why this is filed
rather than squeezed into the survey row.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with src/vllm/model_executor/models/ltx2_text_encoder.cpp:1166-1187 and Gemma4Model::ForwardBody in src/vllm/model_executor/models/gemma4.cpp:733-756. Inspect the existing model tests and tests/vllm/models/ltx2_gemma4_text_config.json, then determine how to instrument the LTX call site without changing conditioning. Done means a test verifies the reduced logits shape and demonstrates that the unused lm_head work is avoided.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
machine-learning, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.