Fused attention overflows the stack above K=512, and the failure names nothing

scripts/run_flash_bench.sh at its default sweep dies at every sequence length above 512. The cause is stack exhaustion; the symptom is an exit code with no diagnostic attached to it.

Symptom

K=512    rep 1     0.992s
K=512    rep 2     0.987s
K=1024 rep 1 FAILED (status 1)
[Vx CUDA] no CUDA device (no CUDA-capable device is detected); kernels will run on the host

Program exited with non-zero code: -1

No backtrace, no signal name, no message. [JIT] Program exited with code: -1 and nothing else.

That is worth its own attention: the runtime does install a SIGSEGV handler that prints a backtrace, and it fired usefully elsewhere in this same session. It cannot fire here, because a handler needs stack to run and the stack is what ran out. So the one failure mode the handler cannot report is the one that produces no evidence at all.

Cause

Default stack. Raising it fixes every failing size:

ulimit -s 524288        # 512 MB, up from the default 8192 KB

K	8 MB stack	512 MB stack
512	works	works
1024	dies	works, o[0][0] = 5.115
2048	dies	works, o[0][0] = 10.235
4096	dies	works, o[0][0] = 20.475

All four values match the fixture's expectations exactly, and scale linearly in K as they should.

Not memory pressure: peak RSS is 256 MB on a machine with 58 GB, and no OOM kill is recorded.

Why this matters more than the workaround

A fused attention forward at sequence length 1024 is an ordinary input, not an exotic one. The fixture is small -- 128 queries, head dim 64, tile 64 -- so a 16-tile sequence should not be near any limit.

The __VX_NT__ substitution is sk / TILE, so tile count grows linearly with K: 8 tiles at K=512, 64 at K=4096. Something per-tile is being placed in the entry block rather than reused, which is consistent with the note in 501ebbe8 about moving device-side scratch onto the stack. Worth confirming whether the host path does the same thing and whether the allocation is hoisted per tile.

Reproduction

On any machine, with or without a GPU:

./run_flash_bench.sh -k "512 1024" -n 1     # K=1024 fails
ulimit -s 524288 && ./run_flash_bench.sh -k "512 1024" -n 1   # both pass

The generated sources land in flash-evidence/gen/, so a single size can be run directly:

./target/release/vxc flash-evidence/gen/flash_128_1024_64.vx --run

Suggested

Two things, and the second matters even after the first is fixed:

1. Stop growing stack usage with tile count -- reuse one tile's scratch rather than allocating per tile.
2. Make stack exhaustion say so. A guard page probe, or an alternate signal stack (sigaltstack) so the existing handler can still run when the main stack is gone. An exit code of -1 with an empty error stream is indistinguishable from a dozen other faults, and it cost real time here to identify.

Found while taking the first FlashAttention timing (#251, hiraditya/Vx.1#348). The CPU baseline that came out the far side is in the same run: 0.50 GFLOP/s, flat across all four sequence lengths.

Start by reproducing the failure with scripts/run_flash_bench.sh and compare it with the 512 MB stack run. Inspect the generated flash-evidence/gen/flash_128_1024_64.vx entry path, the per-tile scratch allocation, and the existing SIGSEGV handler. Done means larger K values no longer grow stack usage and stack exhaustion produces a diagnostic.