Sample proposal: persistent kernel for tiny-model autoregressive inference (microGPT, ~4k MACs/token)

Motivation

Most existing samples that demonstrate persistent kernels do so for ray traversal, BFS, or cooperative-groups patterns. The autoregressive inference case — where a tiny model (entirely L1-resident on a CPU, fits in shared memory on a GPU) needs to dodge per-step launch overhead — is well-known in industry but isn't represented in cuda-samples as a standalone, copy-pasteable demonstration.

This proposal is to add a single-file CUDA sample that runs a complete tiny-transformer forward pass (Karpathy's microGPT: 4,192 fp32 parameters, ~4,000 MACs per token) inside one persistent kernel — no relaunches, no host roundtrips during the timed window.

What it demonstrates

• Single-warp persistent block that loads weights into shared memory at kernel entry and runs an N-token autoregressive loop entirely on-device.
• Warp-shuffle reductions (__shfl_xor_sync) for RMSNorm — no shared scratch.
• Cooperative matmul / attention / softmax / sampler within one warp, with a clear pattern for "what stays per-thread vs what reduces across the warp".
• Direct comparison to a naïve launch-per-op variant in the same repo, which loses ~22× to the persistent version — concrete numbers showing where launch overhead lives.

Measured performance (DGX Spark / GB10)

implementation                       tok/sec      notes
-----------------------------  --------------  ----------------------------------
Blackwell · cuda persistent          413,603   single warp, single block, 1 SM
Blackwell · cuda fp32 (naive)         19,127   ~25 launches per token + token-id roundtrip
TALOS-V2 (FPGA, 56MHz)                53,000   reference comparison

Recorded on a stock DGX Spark (GB10, driver 580.142, CUDA 13.0, -arch=sm_121). Reproducer included.

Reference implementation

Working single-file implementation:

• bench_cuda_persistent.cu: https://github.com/CG-8663/talos-vs-macbook-vs-gx10/blob/main/bench_cuda_persistent.cu (305 lines)
• bench_cuda.cu (naïve baseline for comparison): https://github.com/CG-8663/talos-vs-macbook-vs-gx10/blob/main/bench_cuda.cu (301 lines)

Cycle-counting walkthrough: https://github.com/CG-8663/talos-vs-macbook-vs-gx10#why-is-blackwell-slower-than-grace

The wider benchmark suite (Apple silicon comparisons, FPGA reference) lives in an open PR upstream: https://github.com/AlexCheema/talos-vs-macbook/pull/2

What I'd contribute

A Samples/3_CUDA_Features/ (or wherever you'd prefer) entry containing:

1. The persistent-kernel implementation (cleaned up to cuda-samples conventions — Makefile.config, header/source split, etc.).
2. The naïve launch-per-op variant alongside, so the launch-overhead delta is obvious.
3. A small README explaining the pattern, when it's the right tool, and when it isn't (single-stream char-by-char only — multi-stream batched throughput is a different shape).
4. Pre-trained weights (4,192 fp32 = 16 KB) checked in, no external download needed.

Happy to do the work in a PR if this lands as accepted scope.

Why this fits cuda-samples

• Single-file, self-contained. No framework deps past cuda_runtime.h.
• Pedagogical. The naïve-vs-persistent comparison is concrete: same forward pass, 22× speedup from the kernel-fusion change alone.
• Real workload. Karpathy's microGPT is widely recognized; this isn't a synthetic benchmark.
• Modern. Targets Blackwell sm_121 but the pattern is portable back to Volta+.

Open questions for the cuda-samples team

1. Is this scope something you'd accept upstream, or would it fit better as a CUDA-related external sample?
2. Preferred directory placement?
3. Any conventions I should follow that aren't documented in CONTRIBUTING.md?

Thanks for the time.

Start by reviewing the referenced bench_cuda_persistent.cu and bench_cuda.cu implementations, then check CONTRIBUTING.md and the proposed Samples/3_CUDA_Features placement. Confirm the cuda-samples team accepts the scope and conventions before adapting the persistent and naïve variants. Done means a self-contained sample, checked-in weights, README, and reproducible comparison.