conditioning.connector is 23.6% of an LTX-2.5 render and nothing says whether that is the weights or the arithmetic
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 423
- Forks
- 53
- Avg merge
- 20h 26m
- Merged PRs (30d)
- 310
Description
LTX25-RENDER-SPEED-PARITY (#2296) measured an LTX-2.5 render of the oracle's own request at 5.53x the pinned oracle (n = 3 on dgx:gpu0, rc job 0baa109c-43ff-475e-abf1-7a50152ffd5d), and found that the 21B DiT is 2.92% of it. generate.guiders + conditioning.connector are 312.4 s, 60.3% of the wall, and 3.33x the oracle's entire process. An independent instrument sharing no code with the phase log found the GPU idle for 87 to 88% of the render, median utilization 0%.
That row measures and repairs nothing, and its ## Owed says so:
conditioning.connectorandgenerate.guidersare measured and unowned by any row that could move them. #1269 owns the tower's CPU pinning and is the closest existing owner; it does not carry the connector, and it does not carry that all four passes are two duplicated halves.
The blame is not established, and that is what this issue is for. #2296 names a hypothesis and is careful to call it one: that conditioning.connector's 122.388 s is dominated by loading and widening the connector out of the 42 GB DiT file rather than by the connector's arithmetic. Its corroboration is a MEMORY step and not a TIME one -- host peak rises 8.59 GiB across the tower->connector boundary against 8.06 GB predicted for widening 2.016 B parameters to f32. That is evidence the widen HAPPENS, not evidence it is where the seconds went.
The arithmetic points the other way. The connector is a transformer over rows = 1024 (#1269 records the constant) at inner_dim 4096 video / 2048 audio, 8 layers, 12 dim^2 parameters a layer: about 4.2 TFLOP of f32 GEMM per RunConnector call, which at 122.388 s is 34 GFLOP/s. That is a believable-but-poor rate for vt::MatmulBT's threaded CPU arm on 20 cores, and it is also within an order of magnitude of what an 8 GB bf16->f32 widen plus 4 GB of first-touch page faults costs. Neither estimate excludes the other, and the two answers are two different repairs: caching weights, or moving arithmetic.
src/vllm/multimodal/ltx2_video.cpp's RunConnector emits ONE leaf today, so the table cannot separate them. This issue owns the split -- <prefix>.weights around the two Ltx2LoadConnectorWeights calls and <prefix>.compute around Ltx2ConnectorCreateEmbeddings, plus guiders.tower, which splits the other undecomposed leaf (#2296 read 39.3 s out of generate.guiders by subtraction and called it "a reading rather than a measurement") -- and the repair the split blames, with the absolute-quality gate re-run and a same-arm pixel comparison beside it.
Not #1269. That issue owns the Gemma-4 tower, which is host-resident BY TYPE (Gemma4Weights holds OwnedTensor over OwnedBytes, which carries no device field), so moving it is a weight-arm port and not a queue swap. The connector is a different module with different weights and #1269 does not carry it.
Spec: .agents/specs/ltx25-text-cond-device.md. Row LTX25-TEXT-COND-DEVICE.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with src/vllm/multimodal/ltx2_video.cpp and inspect RunConnector, especially the Ltx2LoadConnectorWeights and Ltx2ConnectorCreateEmbeddings calls. Read .agents/specs/ltx25-text-cond-device.md and the LTX25-TEXT-COND-DEVICE row, then compare the new weights, compute, and guiders.tower timings against the absolute-quality gate and same-arm pixel comparison.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100