qwen4_exp QSA cooperative_topk cannot launch on NVIDIA Thor (sm_110): cluster misconfiguration

Row: MODEL-MM-QWEN4-EXP

Found while measuring the run half of e126687a9a for
#2611. Evidence:
.agents/sync/2026-09-03-e126687-runhalf.md §6.

What happens

At upstream vLLM e126687a9a828d513c01a07cd69f025f27d63280, built from source
for TORCH_CUDA_ARCH_LIST=11.0 and run on thor:gpu0 (NVIDIA Thor, compute
capability 11.0, driver 595.78), Qwen4ExpForConditionalGeneration builds,
allocates and reaches a forward pass, then dies in the QSA indexer:

File ".../vllm/models/qwen4_exp/nvidia/indexer_qsa.py", line 343, in forward
    return qsa_select_paged_tokens(
File ".../vllm/models/qwen4_exp/nvidia/ops/qsa.py", line 799, in qsa_select_paged_tokens
    topk_op(logits, visible_blocks, blocks, topk_workspace, block_topk, columns)
RuntimeError: launch_cooperative_cluster, csrc/libtorch_stable/cooperative_topk.cu:46,
  cooperative_topk launch failed: a kernel launch error has occurred due to cluster misconfiguration

The kernel is compiled for this device — CMakeLists.txt lists 11.0f in
COOPERATIVE_TOPK_ARCHS. What fails is the thread-block cluster launch.

What the source says

csrc/libtorch_stable/cooperative_topk.cu launches
clusterDim = {1, CS, 1} where CS is a compile-time template parameter of 4,
8 or 16, and sets cudaFuncAttributeNonPortableClusterSizeAllowed for CS > 8
— upstream saying in its own code that the large size is not portable. The arm
is chosen by a compute-capability major used as a proxy for a cluster-size
capability:

const bool supports_cluster16 = get_device_prop()->major >= 10;
if (num_rows <= 4 && supports_cluster16) {
  launch_cooperative_cluster<TopK, 16>(params, ct::kSmemSize8, stream);
} else if (num_rows <= 8) {
  launch_cooperative_cluster<TopK, 8>(params, ct::kSmemSize8, stream);
} else {
  launch_cooperative_cluster<TopK, 4>(params, ct::kSmemSize4, stream);
}

major >= 10 was written for 10.x datacentre parts and admits Thor's 11.0.

What is NOT established

The obvious hypothesis — that the 16-CTA arm is the whole story — was tested
and not confirmed. A second leg with a larger request batch, intended to
select the 8-CTA arm, failed with the identical error. That A/B is inconclusive
rather than negative: num_rows at that call site is the number of scoring rows,
not the request batch, so the knob did not move the variable it was chosen to
move. Which cluster size was attempted is unknown, and Thor's actual maximum
cluster size was never queried. Both need a lease.

Owed

1. Query Thor's maximum cluster size directly, and launch a trivial kernel at
   sizes 4, 8 and 16 to find where it refuses. That turns the localisation into
   a measurement.
2. Drive num_rows at the cooperative_topk call site deliberately, so the
   three arms can be told apart.
3. Decide what this means for MODEL-MM-QWEN4-EXP's oracle. If upstream cannot
   run qwen4_exp on the only fleet device whose container exposes a GPU, that
   row has no vLLM oracle for its own model on this fleet regardless of the pin,
   and the secondary-oracle table's llama-cpp-qwen4exp entry carries more
   weight than the records currently say.

Caveats on the reproducer

The config was the published Qwen/Qwen3.8-Flash-Next config.json reduced in
depth and count only (48 layers to 4, 512 experts to 8, ngram_vocab_size_base
20,000,000 to 20,000) with load_format="dummy", because every published
safetensors arm exceeds the largest fleet box and upstream's own
tests/models/registry.py marks all three Qwen4Exp architectures
is_available_online=False. Width was left alone. The cluster size is a
compile-time constant, so no part of that reduction could have selected it.

Start with csrc/libtorch_stable/cooperative_topk.cu and the qsa_select_paged_tokens call in csrc/libtorch_stable/ops/qsa.cu; review evidence/.agents/sync/2026-09-03-e126687-runhalf.md §6. Query Thor's maximum cluster size, test launches at 4, 8, and 16, and drive num_rows deliberately to identify the selected arm. Done means the failing cluster size and the resulting Qwen4Exp oracle status are recorded.