E2E: model_is_ready readiness check can pass on a stale Lemonade router model (chat-06/serve-07)

Summary

While investigating a Strix Halo Windows E2E failure on the new DevLab Dispatch pool lane (see PR #377), found a pre-existing race in the E2E harness's serve-readiness check that isn't specific to the pool — the pool's slower cold-start timing just made it reproduce reliably (2/2 runs).

What happens

chat-06 ("Chat requests that include tool definitions are accepted") and serve-07 ("A model served on lemonade responds to inference requests") both request a bare model name (Qwen3-0.6B-GGUF, no owner/repo). On Windows this routes through Lemonade's long-lived multi-model router (spawn_lemond, engines/lemonade/src/lib.rs:614-736) rather than a fresh single-purpose llama-server process (which is what a canonical owner/repo:variant ref like serve-08 always gets, on any OS, via serve_direct_llama_server, lib.rs:597).

The harness's own readiness gate, model_is_ready (tests/e2e-cucumber/tests/e2e/serving_steps.rs:50-70), is a raw substring check on the full /v1/models response body (body.contains(model)). A router that lists a full catalog of known/registered models can satisfy this check while the model it currently has loaded is still whatever an earlier scenario left active. Observed failure mode: the harness proceeds because /v1/models mentions Qwen3, but the router's active model is still Gemma-4-E4B-it-GGUF from a prior scenario, so the chat/inference request returns:

{"error":{"message":"No model loaded: Gemma-4-E4B-it-GGUF","type":"model_not_loaded"}}

The CLI's own internal readiness check is already stricter — models_payload_has_ready_model (engines/lemonade/src/lib.rs:3445-3459) requires field + backend match, and separately does a real chat-completion smoke test (lib.rs:3072, 3544-3552). The E2E harness's model_is_ready doesn't reuse that stricter check.

There's also a comment in serving_steps.rs:43-49 acknowledging the harness added ensure_serve_port_free/kill_listeners_on_port specifically because a leaked prior-scenario serve can still answer on the shared port — this readiness gap is adjacent to that same class of problem.

Why this matters beyond the pool

This is a latent bug that could in principle hit any host running this suite — it's just far rarer on hardware with a warm model cache, where downloads are near-instant and the race window is tiny. On the ephemeral DevLab pool (cold download every run, no warm cache), the window is wide enough to hit reliably (2/2 runs).

Suggested fix

Tighten model_is_ready (serving_steps.rs:50-70) to check actual-loaded-model state rather than substring-matching the full /v1/models listing — e.g. reuse (or call into) models_payload_has_ready_model's stricter logic, or require a successful chat-completion probe before considering the model ready.

Separately (lower priority, more invasive): Lemonade's engine never calls terminate_process_tree when tearing down a serve (rocm-cli's own stop path uses a single-PID kill via terminate_pid, lib.rs:3824-3826), unlike vLLM's engine which does (engines/vllm/src/lib.rs:895). terminate_process_tree is also a documented no-op on Windows (crates/rocm-core/src/lib.rs:1384-1391), so if lemond spawns a child backend process, Windows has no job-object/tree-kill to reap it. Worth confirming whether this contributes before investing in it.

Reproduction

Dispatch e2e-selfhosted.yml with platform: strix-windows against the DevLab pool (see #376/#377) — reproduced identically on 2 separate runs:

• https://github.com/ROCm/rocm-cli/actions/runs/34447923066
• https://github.com/ROCm/rocm-cli/actions/runs/34449549298

Start in tests/e2e-cucumber/tests/e2e/serving_steps.rs, especially model_is_ready, and compare its /v1/models check with models_payload_has_ready_model in engines/lemonade/src/lib.rs:3445-3459. Reproduce with the strix-windows e2e-selfhosted.yml lane or inspect the cited chat-06 and serve-07 scenarios. Done means readiness reflects the currently loaded requested model and the affected E2E checks no longer proceed against a stale router model.