fix(TOOLS-STRUCTURED-CORE): the grammar matcher rejects every token after termination, so a multi-token batch kills its own request

Row: TOOLS-STRUCTURED-CORE (.agents/engine-matrix.md:160); the parity claim that must move with it is owned by TOOLS-XGRAMMAR (.agents/engine-matrix.md:161).

Found by wave PORTQ-5 (#2679) re-deriving PORT-NOW entry [197], upstream 12f64b39d2 vllm#52805. Nothing was executed — this is a static reading of source.

What upstream did

XgrammarGrammar.accept_tokens returns True (was False) when already terminated and breaks out of the loop the moment the matcher terminates mid-batch, instead of only discovering it after the whole batch; validate_tokens returns [] when terminated and breaks at the terminating token; reset() clears _is_terminated. The motivating case is MTP / spec-decode batches.

What this tree does

NativeGrammar::accept_tokens (src/vllm/v1/structured_output/backend_native.cpp:1139-1147) advances over every token and returns false on the first failure, with no post-termination early-out in either direction:

for (const int32_t token : tokens) {
  Snapshot next = history_.back();
  if (!advance_snapshot(next, token)) return false;
  history_.push_back(std::move(next));
}

Once EOS is consumed, cur.done is set, and the comment at :1210-1212 states the consequence outright: accept_tokens returns false in the done state for any token.

The multi-token caller is unguarded (src/vllm/v1/core/sched/scheduler.cpp:1286):

if (!struct_output_request->grammar->accept_tokens(req_id, new_token_ids)) {
  request->status = RequestStatus::kFinishedError;
  ...
}

So a batch whose grammar terminates before its last token fails the request with FINISHED_ERROR — a worse form of the bug upstream fixed. StructuredOutputManager guards its own per-token calls (manager.cpp:113), which is why only the scheduler path is exposed.

This lands in the native matcher, not in backend_xgrammar.cpp: XgrammarStructuredOutputBackend::compile_grammar (src/vllm/v1/structured_output/backend_xgrammar.cpp:27-58) delegates to inner_, so both backends share one matcher.

Two of the three sub-changes are already satisfied

NativeGrammar::reset() (:1183-1190) rebuilds history_ from InitState, so there is no stale flag to clear. validate_tokens (:1149-1158) has no production caller — the only references outside structured_output/ are the two deferral comments at scheduler.cpp:1540,1559.

This tree is at the pin

git show 5559679229:vllm/v1/structured_output/backend_xgrammar.py carries _is_terminated, the return False guard, the unguarded validate_tokens and a reset() that does not clear the flag. Not a pre-pin hole.

Size and a caution for the implementer

~15 lines of product code plus a red-first test driving a terminating batch through Scheduler::update_from_output.

Do not key the early-out on is_terminated(). That predicate also returns true for cur.awaiting — a lazy grammar that has not yet seen its trigger (backend_native.cpp:1167-1177). An unconditional "terminated => accept everything and stop" would silently disable constraint enforcement for every lazy / structural-tag grammar. Key it on cur.done / IsFullyMatched.

Record edit owed in the same change

.agents/specs/xgrammar-backend.md:132 asserts the matcher is "already 1:1 with backend_types.py" on accept/validate/reset — a claim of parity with the pre-commit behaviour, which this commit moves.

Not established

That a spec-decode batch reaches accept_tokens with a post-EOS token in a shipped configuration. Nothing was run.

Start with NativeGrammar::accept_tokens, validate_tokens, and reset() in src/vllm/v1/structured_output/backend_native.cpp, then trace the scheduler call at src/vllm/v1/core/sched/scheduler.cpp:1286 and update_from_output. Add a red-first test for a terminating multi-token batch while preserving lazy grammar enforcement, and update .agents/specs/xgrammar-backend.md:132 when the behavior and parity claim are correct.