Provision a systemd user environment for generic GPU E2E
- Dominant language
- TypeScript
- Stars
- 22.5k
- Forks
- 3.1k
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 715
Description
## Investigation summary
- The generic NVIDIA GPU runner validates the exact source revision and GPU host before the failure. The GPU is not the failing component.
- PR #9726 commit `6b19a0c2793094bae62ad1a8dbcd58971931e315` preserves the supported standalone installer path when `/run/user/` is absent and adds boundary coverage for that case.
- The real installer then finds an enabled, noncanonical user-service activation link while no same-user systemd manager is reachable.
- Without the manager, NemoClaw cannot qualify that service's executable, arguments, ownership, hooks, activation state, or port. The installer correctly fails closed under the service-identity contract from #9705.
- The safe repair belongs to runner provisioning: provide an inspectable same-user manager or a clean ephemeral runner account. Do not weaken the installer or change foreign services from the workflow.
## Description
The self-hosted runner labeled `linux-amd64-gpu-rtxpro6000-latest-1` cannot complete the generic NVIDIA GPU E2E used to qualify PR #9726.
The test runs the complete NemoClaw installer before GPU inference. The runner currently has both of these conditions:
1. `/run/user/` and its same-user systemd bus are unavailable.
2. At least one enabled, noncanonical user-service activation link is visible to the installer.
This leaves the installer able to prove that an unknown service may start later, but unable to inspect whether that service can claim the selected gateway port. The expected environment is either:
- an already-running, same-user systemd manager whose service state and unit path can be inspected; or
- a clean ephemeral runner account with no enabled noncanonical user-service activation links.
This issue tracks runner provisioning only. Product service policy remains owned by #9705, and PR #9726 remains the linked qualification consumer.
## Reproduction steps
1. Run the `llama.cpp on generic NVIDIA GPU` job for PR #9726 on `linux-amd64-gpu-rtxpro6000-latest-1`.
2. Let the workflow validate the source revision and GPU host.
3. Let the workflow preserve the standalone path because `/run/user/` is absent.
4. Run the complete NemoClaw installer.
5. Observe that installation stops before onboarding because an enabled noncanonical user service cannot be qualified without the user manager.
Evidence:
- [Generic GPU job after the missing-runtime boundary fix](https://github.com/NVIDIA/NemoClaw/actions/runs/32518141313/job/96884309104)
- [Earlier job that stopped at the workflow runtime prerequisite](https://github.com/NVIDIA/NemoClaw/actions/runs/32512531239/job/96866844767)
## Environment
- Repository: `NVIDIA/NemoClaw`
- Related product issue: #9705
- Qualification consumer: PR #9726
- First runner-boundary repair commit: `6b19a0c2793094bae62ad1a8dbcd58971931e315`
- Current PR commit containing that repair and current `main`: `14758ed9f42ffd648848f47ec55380d00c7607b3`
- Runner label: `linux-amd64-gpu-rtxpro6000-latest-1`
- Runtime state: `/run/user/` absent; enabled noncanonical user-service activation present
## Debug output
No NemoClaw debug bundle is available because the complete installer stops before onboarding registers the gateway. The job log records the fixed diagnostic:
```text
The systemd user manager is unavailable, and a noncanonical enabled user service cannot be qualified for selected port 8080.
```
The later `Unknown gateway 'nemoclaw'` cleanup message is a consequence of installation stopping before gateway registration.
## Required runner contract
If this remains a persistent self-hosted account, the runner owner must provision an already-running same-user systemd manager with all of these properties:
- `/run/user/` is a non-symlink directory owned by the runner UID and GID with mode `0700`.
- `/run/user//bus` is a non-symlink socket owned by the runner UID.
- With `DBUS_SESSION_BUS_ADDRESS` unset and `XDG_RUNTIME_DIR=/run/user/`, the existing read-only active-service query succeeds.
- The exact `org.freedesktop.systemd1.Manager.UnitPath` D-Bus property query succeeds.
- Every enabled user-service activation root reported through `UnitPath` is readable and inspectable by the runner account.
- The runner owner audits and owns the lifecycle of enabled user services on this account.
The preferred alternative is a clean ephemeral runner or account that has no enabled noncanonical user-service activation links.
## Security constraints
- Do not start an unmanaged user manager inside the workflow. Starting it can activate persistent, unknown enabled units and cannot guarantee cleanup after cancellation or runner loss.
- Do not delete, disable, rename, filter, or ignore the foreign activation from the workflow.
- Do not add a retry or timeout for this deterministic environment state.
- Do not relax the installer fail-closed behavior from #9705.
- Keep manager probes read-only, suppress their output, and persist only the verified `XDG_RUNTIME_DIR` value.
## Acceptance criteria
- The runner satisfies the runtime-directory, bus-socket, active-service, and `Manager.UnitPath` checks already present in `.github/workflows/pr-self-hosted.yaml`, or the job uses a clean ephemeral account with no competing activation links.
- The generic NVIDIA GPU job reaches and completes the complete installer and inference qualification at one exact PR commit.
- The service-identity diagnostic above does not occur because every activation candidate is inspectable or absent.
- Cleanup and artifact upload complete without the workflow making persistent service changes on the runner.
- Runner ownership, provisioning, and cleanup expectations are documented in the repository-owned E2E guidance.
## Out of scope
- Changing the service-identity policy in product code
- Creating a user manager from the test workflow
- Removing or modifying foreign runner services from the test workflow
- Adding retries or longer timeouts
## Checklist
- [x] I confirmed this bug is reproducible
- [x] I searched existing issues and this is not a duplicate
Contributor guide
Research direction
Read .github/workflows/pr-self-hosted.yaml and run the generic NVIDIA GPU job for PR #9726 on the named runner. Verify the runtime directory, bus, active-service, and Manager.UnitPath checks, then confirm the complete installer and inference qualification finish without persistent service changes. Document runner ownership, provisioning, and cleanup expectations in the repository-owned E2E guidance.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, linux
- Domain
- ci-cd, devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100