The road to peak attention: prefill GFLOP/s and decode GB/s, measured against their rooflines (#251 campaign)

Tracking issue for the performance campaign on the hiraditya/Vx.1#251 emission path, started 2026-08-20 after grid-striding landed (fe408e57) and the A100 SQ-scaling sweep confirmed ~99% scaling efficiency to 64 SMs with the closed form exact at every grid size.

Two regimes, two rooflines

Attention has two shapes with different physics, and one number cannot represent both:

Every claimed improvement reports against its roofline, from device-side events (VX_TIME_KERNEL, 47593f4a -- wall clock on rented pods fit a negative slope from jitter), with the closed-form check passing (o = 0.01*(SK-1)/2 under the zero-query fill) and, for kernel changes, the flash_attention_placed_verified.vx EXPECT row reproduced through the 1x1 launcher.

Where we stand (measured, A100-SXM4-80GB)

Prefill -- scripts/run_flash_bench.sh, fp32, one query per thread:

Scaling to 64 blocks is ~ideal; the cliff at 109 blocks is throughput quantization, not occupancy (driver reports 16 blocks/SM co-resident; one 4-warp block already saturates a per-SM issue pipe). Diagnosis: scalar loads, one element at a time -- the kernel is load-issue-bound, with K/V L2-resident at ~145 GB/s effective.

Reference points on the same box: cuBLAS SGEMM 16223 GFLOP/s (83% of FP32 peak); FlashAttention-2 reaches 50-73% of the 312 TF/s FP16 tensor-core peak (~225 TF/s) on A100.

Decode -- did not exist until this issue. scripts/templates/flash_decode_bench.vx (in review): flash-decoding's split-K written as two placed Vx regions -- partials owned by (head, chunk), combine owned by head -- both admitted by the hiraditya/Vx.1#251 disjointness rule as plain source, no runtime additions. Baseline being measured now; pre-registered expectation: C=1 at H=8 is 8 threads reading ~0.02% of HBM peak, and scaling with H*C should track the prefill block-scaling curve until per-SM issue saturates again.

The ladder

Ordered by measured diagnosis, not by folklore; each rung gets numbers before the next starts.

• R1 -- wider loads, more work per thread (both regimes). The measured bottleneck. float4-shaped access needs the emitter to vectorize the inner d loops or the language to express slice loads on the device path.
• R2 -- SMEM tiles for K/V (prefill). Stage tiles once per block instead of every thread re-reading through L2. Space-3 alloca machinery already exists (materializeGpuKernels rewrites them to .shared globals).
• R3 -- cooperative queries (prefill): partition a query block across warps, threads cooperate on one row's softmax instead of one thread owning it. This is FA-2's work-partitioning insight and requires cross-thread reduction within a block (barriers exist: 0 used today).
• R4 -- decode split-K tuning: chunk count vs SK, the combine's cost at large C, and whether region 2 merges into a tree.
• R5 -- f16/bf16 (#320): halves the bytes decode must read (its roofline doubles in tokens/s) and unlocks tensor cores for prefill.
• R6 -- MMA / tensor cores (prefill): the 16x that separates CUDA-core fp32 from the FA-2 comparison. Blocked on R5.

Milestone gates, so the campaign has falsifiable exits:

• prefill >= 1 TF/s fp32 (5% of peak) with R1+R2;
• decode >= 10% of HBM roofline at SK=16k;
• then re-aim at the vLLM cost-of-admission comparison from the hiraditya/Vx.1#321 plan.

Discipline

• Pre-register before measuring (the SQ-scaling prereg caught two wrong mechanisms this week: "bandwidth-bound" was issue-bound, "residency wall" was throughput quantization).
• Rented pods refuse performance counters (ERR_NVGPUCTRPERM); everything must be measurable with events + occupancy queries + closed forms.
• Correctness gates every rung: closed form at every config, EXPECT row through the launcher, full suite on commit.

Start by running scripts/run_flash_bench.sh and reviewing scripts/templates/flash_decode_bench.vx, then trace materializeGpuKernels and the grid-striding change fe408e57. The campaign is complete only when each selected rung has device-side measurements, closed-form checks, the flash_attention_placed_verified.vx EXPECT row through the 1x1 launcher, and the stated milestone gates.