port(TOOLS-STRUCTURED-CORE): a grammar masks the tokenizer EOS only, so a request stop token can end a structured generation mid-grammar

Row: TOOLS-STRUCTURED-CORE

Found by wave PORTQ-4 (#2680)
re-deriving PORT-NOW entry 124 of 5559679229..e126687a9a, upstream
05f0a80016 vllm#49227,
against this tree.

What is wrong

A grammar masks the tokenizer EOS only, never the request's own stop tokens,
so a request with stop_token_ids can terminate a structured-output generation
before the grammar reaches an accepting state.

The masking mechanism is here and is correct:

src/vllm/v1/structured_output/backend_native.cpp:1258-1261
  // EOS/stop tokens are allowed iff the grammar is at an accepting state.
  if (IsAccepting(cur.state)) {
    for (const int32_t sid : shared_->stop_token_ids) set_bit(sid);
  }

What is missing is the plumbing. shared_->stop_token_ids is engine-wide and
fixed at backend construction (backend_native.cpp:1314-1325), and the only
production wiring passes no stop ids at all — MakeNativeBackendFactory(tokenizer_, vocab_size) at src/vllm/entrypoints/model_loader.cpp:2228-2229 leaves the third
parameter defaulted. The compile seam carries no stop parameter on either side:
include/vllm/v1/structured_output/backend_types.h:141-143 is a two-parameter
pure virtual, and src/vllm/v1/structured_output/manager.cpp:41-48 never reads the
request's sampling params.

The source field the fix needs already exists and is already populated:
SamplingParams::all_stop_token_ids at include/vllm/sampling_params.h:263, filled
at src/vllm/sampling_params.cpp:256 and
src/vllm/v1/engine/input_processor.cpp:161-184. Today its only consumer is the
min-tokens masker (src/vllm/v1/worker/gpu/input_batch.cpp:273-279).

Size, and why it is larger than upstream's nine lines

Add a stop-id parameter to the compile_grammar virtual and both overrides, read
all_stop_token_ids in manager.cpp, and move the stop set from
NativeBackendShared to per-NativeGrammar state.

One interaction needs a decision. This tree also excludes stop tokens from the
shared token-byte trie at construction (backend_native.cpp:1327-1340). A
per-request stop id that is in the trie would still be matchable mid-derivation, so
the exclusion cannot stay construction-time if the set becomes per-request. Estimate
~60-120 lines over five files plus a red-first test.

Adjacent, not this issue

.agents/engine-matrix.md:161 lists model_loader.cpp production wiring as a
TOOLS-XGRAMMAR W2 residual, and model_loader.cpp:2228 confirms production still
builds the native backend. So auto to xgrammar selection is real code nothing
served reaches. Whoever ports this will meet that; it is a pre-existing reachability
gap and belongs to that row.

Verification state

Nothing was executed — no build, no test run, no GPU, no lease. Every path:line
above was printed from the tree at a700e8da6 and re-read by the wave operator.
05f0a80016 is not an ancestor of the pin, and compile_grammar at the pin took
(request_type, grammar_spec) only, which is exactly what this tree mirrors — so
this is a genuine post-pin forward-port, not a pre-pin hole.

Start with compile_grammar in include/vllm/v1/structured_output/backend_types.h and its implementations, then trace request sampling params through src/vllm/v1/structured_output/manager.cpp. Inspect all_stop_token_ids and the NativeBackendShared trie setup in backend_native.cpp; done means request stop tokens are handled per request and cannot end generation before an accepting grammar state.