serve-hf-checkpoint-inference fails on the self-hosted lemonade lanes (split into three issues)

This issue has been split. Investigation established that the original report
covered three distinct problems with three different causes. Each now has its own
issue; work and discussion should go there:

• #258 — the Lemonade embeddable requires GLIBC_2.38, so the lemonade
  engine cannot start on Ubuntu 22.04 at all. User-facing, not CI-specific.
• #259 — lemond cannot resolve a writable runtime directory when
  XDG_RUNTIME_DIR is unset, which is the case for any headless invocation. This is
  the blocker the MI300X lane hits today, after its runner moved to Ubuntu 24.04.
• #260 — on Strix Halo Windows, rocm serve owner/repo:variant exits 0 and
  never serves the model. A different symptom from the Linux side and unrelated to
  either cause above.

The GLIBC finding described in the comments below was correct for Ubuntu 22.04 but is
no longer the live cause on the GPU lane — see the A/B comparison across the runner
reimage in the comments. This issue is kept open as the record of how the three were
separated; the original report follows unchanged.

---

Expected behavior

The E2E scenario serve-hf-checkpoint-inference ("14 - A canonical Hugging Face
checkpoint serves and responds to inference",
tests/e2e-cucumber/features/model_serving.feature:79) passes on every self-hosted
lane. It has no xfail row in tests/e2e-cucumber/expectations.toml, so the harness
expects it to pass on any host matching its @requires-gpu @requires-engine:lemonade
tags.

Observed behavior

It fails on two of the three self-hosted lanes, and — worth noting — with two
different failure modes, so this may be two problems rather than one.

E2E tests (GPU) — MI300X

FAILED  And a canonical Hugging Face GGUF checkpoint is being served on lemonade
        Matched: tests/e2e-cucumber/tests/e2e/serving_steps.rs:523:1
        Step panicked. Captured output: rocm serve failed:

rocm serve unsloth/Qwen3-0.6B-GGUF:Q4_0 --engine lemonade --managed exits non-zero.
Note the output after rocm serve failed: is empty — see "Diagnostics gap" below.

E2E tests (Strix Halo, Windows)

FAILED  And a canonical Hugging Face GGUF checkpoint is being served on lemonade
        Step panicked. Captured output: endpoint http://127.0.0.1:11435/v1/models
        did not serve model Qwen3-0.6B within 600s

Here rocm serve returns 0 — the server simply never becomes ready within the 600 s
budget.

E2E tests (Strix Halo, Ubuntu) passes.

Totals are stable across runs: GPU 46 passed / 7 failed (6 xfail, 1 unexpected);
Strix Halo Windows 30 passed / 5 failed (4 xfail, 1 unexpected). The single unexpected
failure is this scenario in both cases.

When it started

The scenario is new. It was added by 37a3b63c ("fix(lemonade): find llama-server under
nested backend dirs", #242), merged 2026-08-12 21:41 UTC — the only commit in history
touching the serve-hf-checkpoint slug.

The last completed GPU run before that merge was clean: 7 xfail, 0 unexpected failures
(run 31607296961, 2026-08-12 14:32 UTC). Every completed run since reports 6 xfail and
this 1 unexpected failure.

Since Ubuntu passes and the other two hosts do not, the reading that fits the evidence
is that the scenario is correctly reporting the fix as incomplete on MI300X and
Windows — but that is inference from the lane results, not something I have confirmed
on the hardware.

Diagnostics gap (blocks root-causing the MI300X side)

The MI300X failure cannot currently be diagnosed from CI, because the step discards the
error text:

let (stdout, _, rc) = crate::run_rocm(world, &["serve", model, "--engine", "lemonade", "--managed"]);
assert!(rc == 0, "rocm serve failed:\n{stdout}");

run_rocm returns (stdout, stderr, rc) (tests/e2e-cucumber/tests/e2e.rs:476), and
the step drops stderr and then prints only stdout — which is empty on a failed
serve. record_command likewise records stdout only, so the E2E report artifact does
not have it either. 11 step definitions share the let (stdout, _, rc) shape.

Including stderr in that assertion message looks like the prerequisite for making any
further progress on the GPU lane.

Environment

• Branch: main
• Workflow: .github/workflows/e2e-selfhosted.yml
• Affected: MI300X GPU lane, Strix Halo (Windows) lane. Strix Halo (Ubuntu) unaffected
• Model: unsloth/Qwen3-0.6B-GGUF:Q4_0, via the owner/repo:variant direct-serve path

Frequency

Always, on both affected lanes, since the scenario was introduced.

Impact

Both lanes are continue-on-error and are not in main's required-status-check list,
so merges are not blocked. The cost is signal: main's self-hosted lanes are
continuously red and therefore red on every open PR, which makes a genuine new
self-hosted regression easy to miss.

Work has been split into #258, #259, and #260, so investigate those issues rather than this record. For the diagnostic prerequisite, start with tests/e2e-cucumber/tests/e2e.rs:476 and the serving step in tests/e2e-cucumber/tests/e2e/serving_steps.rs:523; done means affected E2E failures retain the relevant stderr in their assertion or report output.