mudler / mudler/vllm.cpp

TT backend never advertises SupportsAsyncSampledTokenReadback: async scheduling resolves OFF, test_qwen3_dense_async_serving FATALs on every cached checkpoint

Open
#1,627 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: BACKEND-TENSTORRENT-HOST-FREE-FORWARD

What fails

tests/parity/test_qwen3_dense_async_serving.cpp (the ROW-SERVE-ASYNC-DENSE-MIRROR battery) FATALs on Tenstorrent for every case whose checkpoint is cached on the box — Qwen3-0.6B, Mistral-7B-v0.3, and the 4B case where cached — at the anti-vacuous-pass guard:

REQUIRE(loaded->async_scheduling_enabled());     // tests/parity/test_qwen3_dense_async_serving.cpp:124
REQUIRE(loaded->max_concurrent_batches() == 2);

The engine logs Asynchronous scheduling is disabled (max_concurrent_batches=1) at load. The remaining cases SKIP as checkpoint-absent (llama-3.2-1B, internlm2), so the battery reads 3 FATAL / 5 skip on TT.

Mechanism

LoadedEngine resolves async scheduling from runner_.runner_supports_async() (src/vllm/entrypoints/model_loader.cpp:1598-1605), which the shared runner derives from the backend capability vt::Backend::SupportsAsyncSampledTokenReadback() (src/vllm/v1/worker/gpu/runner.cpp:109-112, declared include/vt/backend.h:186 with a false default). Only the CPU and CUDA backends override it (src/vt/cpu/cpu_backend.cpp:38, the CUDA async_device_mirror() path). The Tenstorrent backend has no override, so on TT runner_supports_async() is false, ResolveAsyncScheduling yields OFF, and the REQUIRE fails exactly as designed: the depth-2 path the gate exists to exercise never engages.

Not caused by the R5 host-free flip (#1604)

Pre-existing: git grep -c SupportsAsyncSampledTokenReadback 52e328789 -- src/vt/tenstorrent/ returns zero hits, and the flip commits (b86e3705f, f85492992) touch only src/vt/tenstorrent/*, src/vllm/platforms/tenstorrent.cpp, and tests/vt/test_tenstorrent_backend.cpp — none of the async resolution path. Captured-vs-eager decode mode is orthogonal to the readback capability; the battery would FATAL identically on the pre-flip default.

Why this is not an in-flow fix

Advertising the capability is a design row, not a one-liner. The async input-combine host-reads the sampled token id between steps (combine_sampled_and_draft_tokens), which is exactly the per-step host touch the TT host-free decode loop exists to avoid; CUDA answers it with a device-mirrored id (async_device_mirror), and TT needs an equivalent decided against the tt-metal allocator/ownership model, plus the #323-class stale-host-ids guard re-proven on device. That needs its own spec, oracle comparison, and perf gates (the win/loss vs host-free is an open question), so it cannot ride the #1604 change that merely tripped over it.

Owed by

Filed from the BACKEND-TENSTORRENT-HOST-FREE-FORWARD row (#1604); listed under ## Owed in .agents/specs/tenstorrent-host-free-forward.md. The enabling work (TT SupportsAsyncSampledTokenReadback + device-mirrored sampled ids + this battery green on TT) is the scope.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with include/vt/backend.h and src/vllm/v1/worker/gpu/runner.cpp, then compare the CPU and CUDA implementations with src/vllm/entrypoints/model_loader.cpp. Determine the Tenstorrent device-mirrored sampled-id design and validate it against the stale-host-ids guard. Done means the capability is correctly advertised, the async path is safe, and tests/parity/test_qwen3_dense_async_serving.cpp passes on the cached checkpoints with the required gates.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, performance, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.