mudler / mudler/vllm.cpp

perf(SPEC-GDN-SEGMENTS): the GDN builder computes context_lens on decode-only batches that never read it

Open Beginner friendly
#2,696 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: SPEC-GDN-SEGMENTS

Found by wave PORTQ-4 (#2680)
re-deriving PORT-NOW entry 150 of 5559679229..e126687a9a, upstream
10b7766a90 vllm#51913
"[Attention] Move context_lens_tensor compute into GDN prefill path".

This is the one entry in the tranche whose recorded PORT-NOW line made a claim
about this tree and was right.
The line says "vllm.cpp mirrors gdn_attn and this
elides that per-step computation on decode-only batches"; it checks out.

What is wrong

GdnMetadataBuilder::build computes context_lens unconditionally at the top:

src/vllm/v1/attention/backends/gdn_attn.cpp:97-104
  // context_lens = num_computed_tokens = seq_lens - query_lens
  const std::vector<int32_t> query_lens = m.naive_query_lens();
  std::vector<int32_t> context_lens(static_cast<size_t>(m.num_reqs), 0);
  for (int r = 0; r < m.num_reqs; ++r) { ... }

Its only two consumers, at :316 and :326, are both inside the
if (num_prefills > 0) block opened at :315 — the exact block upstream moves the
computation into.

The header already states the invariant the source does not honour:
include/vllm/v1/attention/backends/gdn_attn.h:106-110 — "Upstream ONLY populates
it when num_prefills > 0 (the decode path never needs it — a decode request always
continues an existing sequence)".

Size, stated honestly

Move the query_lens fetch and the context_lens loop inside the
num_prefills > 0 block. ~6 lines, one file, plus a decode-only-batch case in
tests/vllm/v1/attention/test_gdn_metadata_builder.cpp.

The win is small. The elided work is one host loop over num_reqs per step,
not per layer. This is a real divergence from upstream and a cheap one; it is filed
so it is not lost, not because it is urgent. It may reasonably ride the next
SPEC-GDN-SEGMENTS change rather than take its own pull request — that row's scope
line already names both files as ones it owns.

Verification state

Nothing was executed — no build, no test run, no GPU, no lease. Both the
unconditional loop and the guarded consumers were printed from the tree at
a700e8da6 and re-read by the wave operator. gdn_attn.py is present at
5559679229 and the pre-fix placement is the pin's, so this is a genuine post-pin
forward-port. No prior triage: git log --all --grep 10b7766a90 is empty.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in src/vllm/v1/attention/backends/gdn_attn.cpp at GdnMetadataBuilder::build and read the invariant in include/vllm/v1/attention/backends/gdn_attn.h. Move the context_lens computation into the existing num_prefills > 0 block, then add or update the decode-only case in tests/vllm/v1/attention/test_gdn_metadata_builder.cpp. Run that test to verify decode-only batches avoid the computation while prefill behavior remains correct.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, performance
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
84/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.