ROCm / ROCm/rocm-cli

rocm serve owner/repo:variant on lemonade exits 0 but never serves the model on Strix Halo Windows

Open
#260 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
40
Forks
9
Avg merge
4d 20h
Merged PRs (30d)
59

Description

Splitting the Windows half out of #247, which noted it "may be two problems rather
than one". The two halves have different symptoms and different failure points, so
they need separate owners. #247 tracks the Linux half; this issue is Windows only.

Symptom

On Windows, serving a Hugging Face GGUF checkpoint by its canonical
owner/repo:variant reference on the lemonade engine appears to succeed and then
never produces a usable endpoint:

rocm serve unsloth/Qwen3-0.6B-GGUF:Q4_0 --engine lemonade --managed

The command exits 0 and reports managed service launched ... readiness: starting.
The model never appears at http://127.0.0.1:11435/v1/models, and after 600 s the
E2E step gives up:

Step panicked. Captured output: endpoint http://127.0.0.1:11435/v1/models
did not serve model Qwen3-0.6B within 600s

Nothing tells the user, or CI, why. A user hitting this sees a serve that claims to
have started and an endpoint that never answers.

What this is not

  • Not a broken host or a flake. In the same job on the same machine, scenario 7
    serves the same underlying checkpoint successfully via the short recipe name
    (Qwen3-0.6B-GGUF), and the ROCm llama.cpp backend installs cleanly
    (llama-b9752-bin-win-rocm-7.13-x64.zip + therock-dist-windows-gfx1151-7.13.0.tar.gz,
    "Backend installed successfully"). 31 of 35 scenarios pass. Re-running does not help —
    the runs below are already attempts 2 and 3.
  • Not either of the Linux causes. The MI300X lane has two stacked blockers, both
    Linux-specific: the Lemonade embeddable's glibc floor (#258) and, once that is
    out of the way, an unresolvable runtime directory (#259). Windows uses a
    different artifact and a different launch path, and its symptom is the opposite one —
    exit 0, not exit 1.
  • Not platform-wide breakage. Strix Halo Ubuntu passes this scenario on the same
    ROCm 7.13.0 / lemonade 11.5.1 pin.
  • Not a model-id mismatch in the readiness probe. The direct path passes
    --alias unsloth/Qwen3-0.6B-GGUF:Q4_0, which contains the Qwen3-0.6B substring the
    probe looks for, so a live server would match.

Where it likely lives

The two paths diverge in the lemonade engine:

  • the short recipe name is served through lemond;
  • owner/repo:variant takes serve_hf_checkpoint (engines/lemonade/src/lib.rs:720),
    which uses lemond only to pull the GGUF and then launches llama-server directly
    via serve_direct_llama_server.

Only the second path fails, on Windows only. That narrows it to direct llama-server
launch, GGUF path resolution in the HF cache, or port binding — but which one is not
currently observable (below).

--managed is what makes the exit code useless here: rocm serve returns as soon as the
service is launched, so anything that goes wrong afterwards lands in the managed-service
log, not in the caller's exit status.

Observability gap — please fix as part of this

The failure is currently undiagnosable from CI, and the same gap hurts users:

  1. No serve output is captured. The step uses the plain wait_for_model
    (tests/e2e-cucumber/tests/e2e/serving_steps.rs:70), which panics with only the
    timeout line. Compare the GPU serve step (setup_gpu_model, same file), which on
    failure dumps stdout, stderr, the service-log tail and device state. The failing
    world dump confirms the loss: cli_output: None, cli_stderr: None, cli_rc: None.
  2. The artifact does not have it either. record_command
    (tests/e2e-cucumber/tests/e2e.rs:528) writes only argv/rc/subcommand/command/
    model/engine to commands.jsonl — no stdout, no stderr.
  3. The managed-service log is never uploaded. It is written inside the scenario's
    isolated temp data dir (.../data/services/<service-id>.log), which is deleted with
    the TempDir; the artifact upload only takes tests/e2e-cucumber/results/.

So the one file that would contain the answer is created and destroyed inside the job.
Giving this step the same diagnostics the GPU serve step already collects — including
the managed-service log tail — is the prerequisite for root-causing it. (#248 fixes the
sibling problem of discarded stderr on non-zero exit; it does not cover this path,
which exits 0.)

Beyond CI: a --managed serve that reports success and then silently dies gives the user
nothing to act on either. Surfacing the service log location, or a readiness failure, in
the CLI itself is worth considering under the same fix.

Reproduction

Scenario serve-hf-checkpoint-inference
(tests/e2e-cucumber/features/model_serving.feature:79,
@requires-gpu @requires-engine:lemonade), added by 37a3b63c (#242). It has no xfail row
in tests/e2e-cucumber/expectations.toml, so it is expected to pass on any matching host.

Observed identically on two separate runs of the Strix Halo Windows lane:

Environment

  • Host: self-hosted Strix Halo Windows runner, gfx1151, Windows 10.0.26200.9168
  • Runtime: release-wheel-gfx1151-7-13-0; lemonade pinned to 11.5.1 (runtime-deps.toml)
  • Workflow: .github/workflows/e2e-selfhosted.yml, lane "E2E tests (Strix Halo, Windows)"
  • Model: unsloth/Qwen3-0.6B-GGUF:Q4_0

Impact

Users on Windows cannot serve a checkpoint by owner/repo:variant on lemonade, and get no
error explaining why. In CI the lane is continue-on-error and not required, so merges are
not blocked; the cost is the same loss of signal described in #247.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with engines/lemonade/src/lib.rs:720 and compare the direct checkpoint path with the existing diagnostics in tests/e2e-cucumber/tests/e2e/serving_steps.rs:70 and setup_gpu_model. Read record_command in tests/e2e-cucumber/tests/e2e.rs:528 and reproduce scenario serve-hf-checkpoint-inference from tests/e2e-cucumber/features/model_serving.feature:79. Done means the Windows failure exposes useful serve output, stderr, and the managed-service log tail instead of only timing out.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
cli, observability-sre, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.