mudler / mudler/vllm.cpp

conditioning.connector is 23.6% of an LTX-2.5 render and nothing says whether that is the weights or the arithmetic

Open
#2,354 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

LTX25-RENDER-SPEED-PARITY (#2296) measured an LTX-2.5 render of the oracle's own request at 5.53x the pinned oracle (n = 3 on dgx:gpu0, rc job 0baa109c-43ff-475e-abf1-7a50152ffd5d), and found that the 21B DiT is 2.92% of it. generate.guiders + conditioning.connector are 312.4 s, 60.3% of the wall, and 3.33x the oracle's entire process. An independent instrument sharing no code with the phase log found the GPU idle for 87 to 88% of the render, median utilization 0%.

That row measures and repairs nothing, and its ## Owed says so:

conditioning.connector and generate.guiders are measured and unowned by any row that could move them. #1269 owns the tower's CPU pinning and is the closest existing owner; it does not carry the connector, and it does not carry that all four passes are two duplicated halves.

The blame is not established, and that is what this issue is for. #2296 names a hypothesis and is careful to call it one: that conditioning.connector's 122.388 s is dominated by loading and widening the connector out of the 42 GB DiT file rather than by the connector's arithmetic. Its corroboration is a MEMORY step and not a TIME one -- host peak rises 8.59 GiB across the tower->connector boundary against 8.06 GB predicted for widening 2.016 B parameters to f32. That is evidence the widen HAPPENS, not evidence it is where the seconds went.

The arithmetic points the other way. The connector is a transformer over rows = 1024 (#1269 records the constant) at inner_dim 4096 video / 2048 audio, 8 layers, 12 dim^2 parameters a layer: about 4.2 TFLOP of f32 GEMM per RunConnector call, which at 122.388 s is 34 GFLOP/s. That is a believable-but-poor rate for vt::MatmulBT's threaded CPU arm on 20 cores, and it is also within an order of magnitude of what an 8 GB bf16->f32 widen plus 4 GB of first-touch page faults costs. Neither estimate excludes the other, and the two answers are two different repairs: caching weights, or moving arithmetic.

src/vllm/multimodal/ltx2_video.cpp's RunConnector emits ONE leaf today, so the table cannot separate them. This issue owns the split -- <prefix>.weights around the two Ltx2LoadConnectorWeights calls and <prefix>.compute around Ltx2ConnectorCreateEmbeddings, plus guiders.tower, which splits the other undecomposed leaf (#2296 read 39.3 s out of generate.guiders by subtraction and called it "a reading rather than a measurement") -- and the repair the split blames, with the absolute-quality gate re-run and a same-arm pixel comparison beside it.

Not #1269. That issue owns the Gemma-4 tower, which is host-resident BY TYPE (Gemma4Weights holds OwnedTensor over OwnedBytes, which carries no device field), so moving it is a weight-arm port and not a queue swap. The connector is a different module with different weights and #1269 does not carry it.

Spec: .agents/specs/ltx25-text-cond-device.md. Row LTX25-TEXT-COND-DEVICE.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with src/vllm/multimodal/ltx2_video.cpp and inspect RunConnector, especially the Ltx2LoadConnectorWeights and Ltx2ConnectorCreateEmbeddings calls. Read .agents/specs/ltx25-text-cond-device.md and the LTX25-TEXT-COND-DEVICE row, then compare the new weights, compute, and guiders.tower timings against the absolute-quality gate and same-arm pixel comparison.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.