Only f32/i32/i64 execute: f16, bf16, i8, i4 are declaration-only — every dtype vLLM serves Llama in

Start in src/codegen/lower/mod.rs at extract_mlir_element_type, especially the scalar/container handling and element-type match. Run the per-dtype Tensor program and inspect the existing Llama fixtures, which currently validate emitted MLIR rather than values. Done means f16, bf16, i8, i16, i4, and unsigned variants execute with correct values, while fp8 retains its tracked fail-closed error.

f32, i32 and i64 are the only numeric element types that survive codegen. Every dtype an LLM is actually served in — f16, bf16, fp8, i8, i4 — can be declared and capacity-checked but cannot be executed.

Found while scoping the GPU demo (#319). It bounds every dtype claim in the MLSys fleet work to admission rather than execution, and it is on the critical path for a Llama/FlashAttention demo whose numbers anyone would want to show.

Measured

Identical program per dtype — declare a Tensor<T>([4,4]), write elements, read one back:

fn main() -> i32 {
  let mut a = Tensor<T>([4, 4]);
  for i in 0..4 { for j in 0..4 { a[i][j] = <lit>; } }
  print(a[0][0]);
  return 0;
}

u8/u4 report through their signed spelling (i8/i4), so the unsigned variants are not separately handled either.

Why declaration works and execution does not

fleet/admit.vx emits memref<?x?xf16> and admits correctly, which is what makes this easy to miss: the type reaches MLIR, so capacity arithmetic over f16 is real and correct. It is element access that fails. admit.vx only declares and allocates — it never reads or writes an element — so the whole fleet-admission matrix is unaffected and its results stand.

There are two distinct mechanisms in extract_mlir_element_type:

1. f16 / bf16 are already in the accept list (lines 155-156) and still fail. They therefore cannot be reaching the match at all: the function early-returns at line 141 for any string that is not wrapped in memref<…> / tensor<…>, and the error text shows a bare f16 arriving. A scalar element type extracted from a load is being handed to a function that only accepts container spellings.
2. i8, i16, i4, u8, u4 are absent from the match entirely (line 154-163), so they fail at line 162 regardless of how they arrive.

Both need fixing; (1) is likely a few lines, since the accept list already names the types.

The fp8 case is not a bug — it is deliberately gated with a message naming hiraditya/Vx.1#249. That is the right behaviour (fail closed, point at the tracker) and only listed here for completeness of the dtype picture.

Why it matters beyond tidiness

• 2× on a bandwidth-bound workload. Batch-1 Llama decode is memory-bandwidth-bound; the GEMMs degenerate to GEMV. Running f32 where vLLM runs f16/bf16 reads twice the bytes per token, which is a straight 2× penalty before any kernel is involved. On a cuBLAS-backed demo this would be the single largest term in any performance delta, and it has nothing to do with kernel quality.
• It rules out the deployment-realistic configurations. AWQ/GPTQ 4-bit is how Llama is most commonly served. A fleet-admission story that can admit an i4 configuration but never execute one has a visible gap between what it reasons about and what it can run.
• cuBLAS is ready for these and we are not. HGEMM / cublasGemmEx cover fp16/bf16/int8 natively, so the blocker is entirely on the Vx side.

Acceptance

• The program above executes and prints the correct value for f16, bf16, i8, i16, i4 (and the u* spellings).
• A corpus fixture per dtype, asserting the value, not just that MLIR was emitted — the existing Llama fixtures only check emit-mlir produces a module, which is how this stayed invisible.
• fp8 continues to fail closed with its hiraditya/Vx.1#249 message rather than silently producing wrong results.
• The distinction stays documented: capacity admission over a dtype does not imply execution in it.

Related

• hiraditya/Vx#319 — GPU demo; this is a Track-A blocker.
• hiraditya/Vx.1#249 — FP8/FP6/FP4 codegen + block scaling.
• hiraditya/Vx.1#280 — added the fp8 element types (declaration side, landed).
• hiraditya/Vx.1#290 — the MLSys readiness table lists f16/bf16 and i4 packing under "already there"; that is accurate for capacity math and should be read as such.