improve external compute-driver startup API and test coverage
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 8.7k
- Forks
- 1.3k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 253
Description
User Story
As an OpenShell operator or contributor, I want standalone compute drivers to share a clear startup contract, so that I can run gateway-external drivers consistently across Docker, Podman, VM, and CI environments.
Problem Statement
OpenShell supports external compute drivers through the gateway remote driver path: the gateway selects a configured driver name and connects to a Unix socket. However, the startup UX for standalone driver processes is not consistent across drivers.
The gateway-side contract is generic:
- select one compute driver
- configure
[openshell.drivers.<driver>].socket_path - connect to the external driver over the socket
The driver-side startup contract is not yet generic:
- Docker uses
--bind-socket,--config, and--gateway-bind - Podman uses
--bind-socketorOPENSHELL_COMPUTE_DRIVER_SOCKETplus many individual flags/env vars - VM has its own set of startup flags
- Kubernetes external-driver mode is expressed through Helm/image composition
This makes external-driver testing and CI artifact reuse harder than it should be. It also makes it unclear which options are part of the common external-driver API versus driver-specific runtime configuration.
Impact / Why This Matters
Today, each E2E harness has to know driver-specific launch details. That increases duplication and makes it easy for coverage to drift. In particular, restoring rootless Podman through Nix VM tooling exposed that the gateway-side external-driver model is generic, but the standalone driver launch UX is not.
This blocks a clean CI workflow where prepared artifacts can be composed predictably:
- driver-free gateway artifact
- standalone compute-driver artifact
- supervisor/sandbox artifact
- Nix VM guest runtime
This also leaves a narrower coverage gap after #2981 removed the original rootless Podman lane: rootless Podman with pasta, a driver-free gateway, and an external openshell-driver-podman process.
Related PR: https://github.com/NVIDIA/OpenShell/pull/2981
Proposed Design
Define and implement a consistent startup UX for standalone compute drivers.
Externally observable behavior should include:
- Every standalone compute driver can bind its public gateway socket through the same option, for example
--bind-socket. - Every standalone compute driver can consume driver runtime configuration through a consistent mechanism, for example
--config. - Gateway callback/listener inputs have a consistent name and meaning across drivers.
- The gateway continues to select external mode through
[openshell.drivers.<driver>].socket_path. - E2E tooling can start an external driver without embedding one-off launch logic for each driver.
Use VM-backed rootless Podman testing as one validation path:
- Build or download a driver-free gateway artifact.
- Build or download
openshell-driver-podman. - Start both inside the Ubuntu 26.04 Nix test guest.
- Run the rootless Podman E2E suite against the external driver.
Acceptance Criteria
- Document the intended external compute-driver startup contract.
- Align standalone driver CLI options where practical.
- Preserve backward-compatible env vars or document any intentional migration.
- Add tests or launch checks proving the common contract works for at least Docker and Podman.
- Add CI artifact support for a driver-free gateway if needed.
- Add CI artifact support for standalone compute-driver binaries if needed.
- Add or update E2E coverage for rootless Podman using an external
openshell-driver-podmaninside the Nix VM. - Update E2E docs with the external-driver workflow.
Alternatives Considered
Keep adding driver-specific branches to each E2E helper. This is simpler short-term, but it preserves the API ambiguity and makes future external-driver coverage harder to compose.
Treat rootless Podman external-driver coverage as only a test gap. That restores one lost CI scenario, but it does not address the underlying inconsistency exposed by the work.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the gateway remote driver path and compare the standalone Docker and Podman startup options, including --bind-socket, --config, --gateway-bind, and the documented environment variables. Then inspect the Nix VM rootless Podman E2E workflow. Done means a documented common startup contract, aligned options with migration notes, and passing Docker, Podman, and external-driver checks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, rust
- Domain
- devops, infrastructure, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100