The served prompt-token count diverges from the client's corpus, and it biases our own throughput UP (refiles #1355)

Row: SPEC-PROMPT-TOKEN-DIVERGENCE

Refiles #1355, which no longer
resolves (GraphQL: Could not resolve to an issue) and so can no longer be the
owner its spec and records point at. .agents/specs/prompt-token-divergence.md
and .agents/benchmark-record.md:25533,25560 both still cite it.

Status

The aarch64 tokenizer hypothesis is FALSIFIED — do not re-run it. Mechanisms
1 and 2 (an aarch64 merge/vocabulary ORDER difference, and a UTF-8 decode
truncation inside Encode) were tested and do not reproduce: byte-identical ids
between x86-64 and aarch64, on the golden corpus and on a combining-mark-dense
corpus in the anomaly's own 74-150-marks-per-line regime, under
-cpu cortex-a53 and under both -fsigned-char and -funsigned-char.
Method and limits in ## What I measured. Mechanism 3 is the leading
candidate: the /v1/completions request-parse segment, from_json(const nlohmann::json&, CompletionRequest&) at protocol.cpp:330, which nothing on
aarch64 reaches because the arm CI job sets VLLM_CPP_SERVER=OFF. The root cause
remains unlocated, which is why this stays open.

The recovered description

Recovered verbatim from the retired issue index,
.agents/completed/issue-index.md:448:

Our server reports 5,942 prompt tokens where vLLM reports 6,144 for the
IDENTICAL generated prompts. Found 2026-08-19 in the raw
vllm bench serve --save-detailed files of the Qwen3.8-27B bf16 re-measure
(#915), both arms driven by the byte-identical client invocation from the same
pinned wheel 0.1.dev1+g555967922, same dataset, same seed. input_lens is
the SERVER-reported length — vllm/benchmarks/lib/endpoint_request_func.py:247
overwrites output.prompt_len from the streamed usage.prompt_tokens — and
reads [915, 931, 1024, 1024, 1024, 1024] for us against [1024] x 6 for vLLM
at c1, with 19 of 48 short (877-941) at c8, byte-identical across all three
reps of each leg. Not the client: _align_prompts_to_server_tokenizer
(vllm/benchmarks/serve.py:74,2041-2044) re-aligns against the server's own
/tokenize and prints WARNING: tokenizer mismatch when it disagrees, and
NEITHER arm printed it, so our /tokenize agreed on 1024 while our
usage.prompt_tokens reported 915 for the same request. output_lens is
[128]xN on both arms in every leg, so the campaign's output-throughput, TPOT
and ITL figures are unaffected; total_token_throughput is affected, our c8
196.10 tok/s being computed over 47,072 input tokens where the intended
workload is 49,152. TWO causes and the artifacts cannot separate them:
under-reported usage, or a genuinely truncated prompt — and the second would
mean the two arms did not run the same workload. A greedy token gate cannot see
either, which is why it survived the gate on this checkpoint.

That row's last sentence on output throughput was superseded five weeks later
by the spec and by .agents/benchmark-record.md:25907-25921, which correct it:
output_throughput is total_output_tokens / duration, and the missing prefill
shortens the wall, so it is biased UP.

Grounding verdict: STILL REAL as a server defect, MISDESCRIBED as a harness bias

Three corrections to how this defect is currently relayed.

1. It is not a harness bias, and the harness has never been able to publish
it. The timed client is vLLM's own unmodified vllm bench serve, faithfully
reporting what our server told it. tools/bench/online_gate.py has refused this
exact symptom since b0acec28c (2026-07-11), five weeks before the campaign:

if record.get("total_input_tokens") != expected * INPUT_LEN:      # :771
    raise HarnessError("total input tokens do not match the frozen corpus")
if any(value != INPUT_LEN for value in input_lens):               # :783
    raise HarnessError("raw input lengths do not all equal 1024")

tools/bench/run_serve_low.py:438 carries the same guard, and
tools/bench/summarize_serve_low.py:100-107,191 routes a validator failure into
binding_eligible: False. The 2026-08-19 campaign ran through bench.sh, a
driver on the benchmark host that this repository has never tracked
(git rev-list --all --count -- '**/bench.sh' is 0), so it was never subject to
these guards at all. So on today's tree the defect is a blocked
measurement, not a silently biased one — which matters directly for the aarch64
dgx benchmark, where the harness will hard-fail rather than flatter us.

2. The spec's first ## Owed item is STALE. It says the aarch64 lane runs no
tokenizer gate. That landed on 2026-08-23 (af320abb2, GATE-CI-AARCH64-COVERAGE
W1b, #1385): .github/workflows/ci.yml:1533-1538 now builds nine tokenizer
targets on ubuntu-24.04-arm, test_tokenizer_parity among them.

3. The gate that closed it is still not sufficient, and this is now measured.
#1385 recorded that judgement; it is confirmed on today's corpus:

	golden corpus	the anomaly regime
combining marks, total	30 over 99 lines	74-150 per prompt
max marks in any one entry	13	74-150
max entry length	109 chars	~1024 tokens

What I measured, and what it excludes

The spec names three surviving mechanisms. Two of them — an aarch64 merge-table
or vocabulary ORDER difference, and a UTF-8 decode truncation inside Encode —
are the aarch64 hypothesis, and the spec says the discriminating test is
examples/tokenize built natively on aarch64. That test no longer needs the
box. CI already installs qemu-user and runs qemu-aarch64 -cpu cortex-a53
(ci.yml:1563-1577), so the same thing cross-builds and runs locally.

Built src/vllm/tokenizer/{tokenizer,bpe,pretokenizer,unicode_data}.cpp at
c796fea41 for x86-64 (g++ 13.3) and for aarch64
(aarch64-linux-gnu-g++ 13.3, -static), against the committed
tests/parity/goldens/tokenizer_qwen36/tokenizer.json (sha256
87a7830d…72de4, matching the recorded oracle), and encoded two corpora.

Instrument precondition: the x86-64 probe reproduces the committed goldens
53/53 entries, ids-for-ids.

corpus	x86-64 vs aarch64	ours vs HF tokenizers 0.22.2
committed golden corpus (1,417 tokens)	identical	53/53
combining-mark-dense, 74-150 marks/line, 2,245 marks, 4,240 tokens	identical	15/20 mismatch
the same corpus NFC-normalised (1,999 marks retained)	identical	20/20

Identical also under -cpu cortex-a53, and under both -fsigned-char and
-funsigned-char, which is the char-signedness hypothesis the arm CI job's own
comment names.

The 15/20 mismatch is the known NFC deviation, not a new defect, and my first
corpus injected it. The mismatching line set equals the non-NFC line set
exactly (15 == 15, set equality). the DEVIATION comment in CheckNormalizer (src/vllm/tokenizer/tokenizer.cpp:510-513)
records that we accept the checkpoint's declared normalizer: NFC and do not
apply it. The campaign's 48 prompts were already NFC (spec item 7), so this is
not the campaign's mechanism — and in the wrong direction anyway: ours came out
longer than HF, where the anomaly is shorter.

So mechanisms 1 and 2 do not reproduce on today's sources. Stated with its
limits, because none of these is closed by this: qemu-user is not real aarch64
silicon; the cross-compiler is not the box's compiler; the golden Qwen3.6
tokenizer.json is not the campaign's Qwen3.8-27B one (that is on the NAS); and
the tokenizer moved 864 lines across those four files since the campaign
build 1dac4f9a7, so a defect that existed then may simply have been repaired by
unrelated work.

What remains

• Mechanism 3, untested and now the leading candidate: the /v1/completions
  request-parse segment — from_json(const nlohmann::json&, CompletionRequest&)
  (protocol.cpp:330, one match in the file) and serving_completion.cpp. Spec item 3 explicitly
  did not exercise it, and the arm CI job sets VLLM_CPP_SERVER=OFF, so nothing
  on aarch64 reaches it.
• The parity corpus is a necessary and not a sufficient probe. LANDED
  in 3b62362dc. It carried 28 combining marks across its 64 entries, at most
  13 in any single entry, against the 74-150 per prompt that produced the
  anomaly. Six NFC-stable entries in that regime were added (Devanagari matras,
  Thai marks, Arabic harakat, 78-149 marks each) and encodings.json regenerated
  with the committed tools/parity/dump_tokenizer.py under the recorded
  tokenizers 0.22.2; all 64 existing entries stayed byte-identical.
  Discrimination measured, not asserted: capping marks at 6 per letter-run in
  MatchLetterRun leaves the old 64 entries green at 1175/1175 assertions and
  fails five of the six new ones. Gate green at 2243/2243 on x86-64 and under
  qemu-aarch64 -cpu cortex-a53, against 1175 before.

Consequence for the "at least as fast as vLLM" claim

Sign is UP, in our favour, and it exceeds the noise on both legs.
Re-derived independently from the raw artifacts, which are not in the tree
but are on the NAS at /mnt/nas_share/rc/q38bf16/out/bench-20260819T035148Z:

	c1	c8
input tokens served / intended	5,942 / 6,144	47,072 / 49,152
output_lens	all 128	all 128
marginal prefill cost	1.027 ms/token (6 points)	1,300-1,800 tok/s first wave
shortfall as wall time	0.207 s of 174.39 s	1.16-1.60 s of 271.4 s
bias on output_throughput	+0.119 %	+0.43 to +0.59 %
CV over 3 reps (population)	0.0390 %	0.2052 %
CV over 3 reps (sample)	0.0478 %	0.2513 %

Both CVs reproduce .agents/benchmark-record.md:25907-25921 exactly. My c1 bias
is 0.119 % against the record's ~0.13 %, the difference being the marginal-cost
estimator (1.027 ms/token from the six short-prompt deltas, against 1.10 from 15
non-outlier points). Either way it is roughly 2.5x the c1 CV and about 2x the c8
CV, so the bias is larger than the precision printed beside it. Both figures
are LOWER bounds, because a shorter context also cheapens every decode step.

The records are honest about it today — the total-token cells are annotated
CORRUPTED and "do not set this beside ours". No published ours-over-vLLM ratio
rests on the affected axes.

Start at the /v1/completions request parser, from_json in protocol.cpp:330, and serving_completion.cpp; compare its prompt-token usage with the /tokenize result for the same request. Use examples/tokenize and the existing aarch64 CI path as reference, then establish the remaining cause and verify that served counts match the client corpus and benchmark guards.