Serving-path gaps blocking prompted llama2 parity: char-level tokenizer, no stop condition, String printing

With the transformer itself now matching llama2.c exactly from BOS (997d3d87, 64/64 token ids identical), the remaining gaps between the Vx port and a real serving path are all outside the forward pass. Recording them together because each one independently blocks a prompted parity comparison, which is what M3's acceptance in hiraditya/Vx.1#321 needs.

1. vx_encode_prompt is character-level, not BPE

stdlib/rust_core/src/ffi/llama.rs:185 looks up each character in the vocab individually, with a byte fallback, and has no merge loop. llama2.c's encode() repeatedly merges the adjacent pair with the best score.

Concretely, llama2.c encodes "faraway" as two tokens (2215, 21694); the Vx encoder emits one token per character. The prompts therefore tokenize differently and no prompted comparison can ever match, no matter how correct the transformer is. This is why the parity check in 997d3d87 deliberately starts from BOS.

It also omits llama2.c's dummy prefix — the leading space added before the prompt text.

Fixing this means porting the merge loop: repeatedly scan adjacent token pairs, look up the concatenation in the vocab, and merge the best-scoring pair until none improves. The tokenizer's merge scores are already loaded.

2. The generation loop has no stop condition

llama2.c breaks on if (next == 1) — BOS doubles as the sequence delimiter. tests/backend/pass/llama2.vx has no such break, so it keeps generating past the end of the story.

This was observed directly: the prompted reference run stops at step 207 when it emits token 1, while the Vx loop would continue to its token budget. Beyond that point the two diverge for reasons unrelated to numerics.

3. llama2_v2.vx prints token pointers instead of text

The idiomatic variant (the one hiraditya/Vx.1#319 recommends as the serving base, since it uses the @ matmul operator and &mut f32 slices rather than raw pointers) compiles and runs, but its output is a stream of large integers rather than text.

Cause: Tokenizer::decode returns a String struct, and printing it emits Warning: unsupported print arg type !llvm.struct<"String", (ptr)> at compile time, then falls through to print_i32 — so it prints the pointer. Either print! should accept a String (calling as_c_str and print_str), or the warning should be an error rather than silently printing something meaningless.

The silent fallthrough is the more serious half: a compile-time warning that produces confidently wrong runtime output is exactly the failure mode hiraditya/Vx.1#321's plan warns about.

4. config.seq_len is taken from the token budget, not the model

llama2.vx sets seq_len: max_tokens from vx_get_llama_config() (default 1000) instead of the model header's 256. The KV-cache indexing stays self-consistent, so it is not a correctness bug, but it over-allocates the cache and lets generation run past the trained context length, which produces out-of-distribution output rather than an error.

Tooling that already exists for this

A patched llama2.c tracer is set up on the EC2 build box at ~/ref/ (make_run_trace.py regenerates it from any upstream run.c and hard-fails if an anchor stops matching). It emits per-step step=<pos> token=<id> lines and first-step logits, and it carries a -T "1,9038,..." flag that injects explicit prompt token IDs, bypassing BPE.

That flag is the shortcut for prompted parity before item 1 is done: feed the C reference the Vx tokenizer's exact token sequence and the two should agree token for token. If they do, the transformer is confirmed correct under prompting too, and the tokenizer is isolated as the only remaining difference.

Related: hiraditya/Vx.1#321 (M0/M3), hiraditya/Vx.1#319.

Start with stdlib/rust_core/src/ffi/llama.rs:185, tests/backend/pass/llama2.vx, and llama2_v2.vx; compare their tokenizer, generation, printing, and config behavior with llama2.c. Use ~/ref/make_run_trace.py and its -T prompt-token option for prompted parity checks. Done means BPE prompting, token-1 stopping, textual String output, and model-derived context length agree with the reference path.