local_assistant_therock_gpu_test.py: direct llama-server mode never detected when service manifest is already ready → false timeout

Current behavior

scripts/local_assistant_therock_gpu_test.py fails every live run against a Lemonade service that uses the direct lemonade-packaged-llama-server backend, even though the service is ready and answering inference. It waits the full --timeout polling /v1/models, then raises:

RuntimeError: managed service endpoint did not report the requested loaded model before timeout: model was not loaded yet; HTTP 200 from /v1/models; endpoint=http://127.0.0.1:11437/v1

Meanwhile the product side is healthy: rocm serve prints readiness: ready, the engine state file has backend_requested: rocm-stable, backend_mode: lemonade-packaged-llama-server, load_response.status: loaded, and the CLI's own inference probe succeeded (inference_verified_at_unix_ms set). A manual POST /v1/chat/completions against the same endpoint returns a correct answer on the GPU.

Expected behavior

The harness should recognize the direct llama-server mode (which the script already has a code path for, lemonade_direct_rocm) and accept the /v1/models entry that names the model, instead of requiring recipe_options.llamacpp_backend, which a bare llama-server never emits.

Steps to reproduce

On a Linux host with a ready managed TheRock runtime and the release binary built:

python scripts/local_assistant_therock_gpu_test.py \
  --rocm target/release/rocm --engine lemonade --model qwen --port 11437 --timeout 90 \
  --runtime-id <active runtime key>

Observed on a6fa76dbca5d767c1190ee47b765d16bad0e4a53 (current main), twice (20-minute and 90-second timeouts). Because the packaged llama-server becomes ready in ~2 s, the service manifest is already status: ready by the time the script reads it, so the bug triggers deterministically.

Root cause (script)

wait_ready_manifest_path (line 395) returns as soon as the service manifest says status: ready, before the branch that loads the engine state file and attaches it as manifest["engine_state"] (lines 397-406). wait_local_endpoint (lines 463-468) then computes

lemonade_direct_rocm = engine == "lemonade" and str(
    engine_state.get("backend_requested") or manifest.get("backend_requested") or ""
).strip().lower().startswith("rocm")

with engine_state == {} and no backend_requested on the service manifest → False → require_rocm_backend=True for the /v1/models probe → payload_entry_reports_rocm_backend never matches the plain llama-server payload:

{"object":"list","data":[{"id":"Qwen3-4B-Instruct-2507-GGUF","aliases":["Qwen3-4B-Instruct-2507-GGUF"],"object":"model","owned_by":"llamacpp","meta":{...}}]}

The engine state file that would have flipped the flag:

{"backend_requested":"rocm-stable","backend_mode":"lemonade-packaged-llama-server","status":"ready","load_response":{"status":"loaded","llamacpp_backend":"rocm-stable"}}

Possible solution

In wait_ready_manifest_path, always load engine_state_path when it exists and attach it to the returned manifest, regardless of whether the service manifest is already ready (i.e. move the engine-state read above the early return, or fall through to it). Alternatively have wait_local_endpoint read engine_state_path itself when manifest["engine_state"] is absent.

Your environment

• rocm-cli channel: nightly (TheRock 10.1.0a20260822, gfx120X-all)
• rocm-cli version (rocm --version): rocm 0.1.0, built from a6fa76dbca5d767c1190ee47b765d16bad0e4a53
• Platform: Linux
• OS / Distro: CachyOS (Arch), kernel 7.2
• GPU/APU: AMD Radeon AI PRO R9700 (gfx1201); second GPU AMD Radeon RX 5700 XT (gfx1010)
• Engine env: lemonade-embeddable-11.5.1, backend rocm-stable, llama-server b9752

Relevant log output

serve plan
  requested model: qwen
  resolved model: Qwen3-4B-Instruct-2507-GGUF
  engine: lemonade
  runtime_id: adopted-nightly-pip-10-1-0a20260822-gfx120x-all-10-1-0a20260822
  selection_source: cli_runtime_id
  device_policy: gpu_required
  gpu: auto (selected 1)
managed service launched
  service_id: lemonade-qwen3-4b-instruct-2507-g-1789163526848
  endpoint: http://127.0.0.1:11437/v1
  readiness: ready
[local-assistant-gpu-test] Stopping managed service lemonade-qwen3-4b-instruct-2507-g-1789163526848.
Local server stopped
  stopped processes: 1
RuntimeError: managed service endpoint did not report the requested loaded model before timeout: model was not loaded yet; HTTP 200 from /v1/models; endpoint=http://127.0.0.1:11437/v1

Service log (product side, same run):

Launching Lemonade packaged rocm-stable llama-server directly: using Lemonade packaged rocm-stable llama-server directly on Linux
0.00.102.614 I   - ROCm0   : AMD Radeon RX 5700 XT (8176 MiB, 8058 MiB free)
0.01.990.717 I srv  llama_server: model loaded
0.01.990.719 I srv  llama_server: server is listening on http://127.0.0.1:11437

Additional context

Side observation from the same run, not the cause of this failure: --gpu auto selected GPU 1 (RX 5700 XT, gfx1010) because it had more free VRAM than the busy gfx1201 card, even though the active runtime family is gfx120X-all and only rocm-sdk-device-gfx1201 is installed. Lemonade's packaged llama-server carries its own gfx1010 code objects so it worked, but the harness has no --gpu passthrough to pin the validation GPU. Related ComfyUI harness issue: #390. Running the documented direct path (rocm serve … --gpu 0) and issuing a chat request manually passed on gfx1201.

Start in scripts/local_assistant_therock_gpu_test.py, especially wait_ready_manifest_path around lines 395-406 and wait_local_endpoint around lines 463-468. Run the documented Linux command with a ready Lemonade direct llama-server service, then verify that the engine state is available before endpoint probing and that the /v1/models entry is accepted without recipe_options.llamacpp_backend.