feat(testing): support independent gateway and supervisor image overrides
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 8.7k
- Forks
- 1.3k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 253
Description
User Story
As an OpenShell developer or downstream integrator, I want to configure the gateway and supervisor test images independently, so that I can run OpenShell’s test wrappers against custom, forked, mirrored, or differently named images.
Problem Statement
OpenShell’s test wrappers generally derive gateway and supervisor images from a shared registry prefix and image tag:
${REGISTRY_VALUE}/gateway:${IMAGE_TAG}
${REGISTRY_VALUE}/supervisor:${IMAGE_TAG}
This assumes that both images use the same registry, use the standard gateway and supervisor repository names, and use the same tag.
That assumption is hard-coded across the Kubernetes, Docker, and Podman test setup paths. It prevents testing images such as:
registry.example.com/custom/openshell-gateway:branch-build
registry.example.com/custom/openshell-sandbox:branch-build
or images that are pinned independently by tag or digest.
Impact / Why This Matters
Users testing downstream builds, private registries, forks, rebuilt artifacts, or alternate image layouts must currently retag their images to match OpenShell’s expected repository names or modify the test scripts locally.
This makes test execution harder to reproduce and creates unnecessary friction for CI systems where gateway and supervisor images may be published under different names or built by different pipelines.
The current Docker and Podman wrappers already support some supervisor-specific overrides, but the behavior is inconsistent across wrappers and there is no equivalent general gateway-image override.
Proposed Design
Add independent gateway and supervisor image overrides to all test wrappers that select or construct OpenShell images.
For example:
GATEWAY_IMAGE="registry.example.com/custom/gateway:test"
SUPERVISOR_IMAGE="registry.example.com/custom/supervisor:test"
When the overrides are not set, preserve the existing registry-and-tag behavior:
GATEWAY_IMAGE="${GATEWAY_IMAGE:-${REGISTRY_VALUE}/gateway:${IMAGE_TAG_VALUE}}"
SUPERVISOR_IMAGE="${SUPERVISOR_IMAGE:-${REGISTRY_VALUE}/supervisor:${IMAGE_TAG_VALUE}}"
The resolved full image references should be used consistently across:
e2e/with-kube-gateway.sh;e2e/with-docker-gateway.sh;e2e/with-podman-gateway.sh;- other test setup wrappers that build, select, or deploy gateway and supervisor images;
- Helm chart values;
- local Docker or Podman image builds;
- k3d and kind image imports;
- external-driver image composition;
- generated gateway configuration; and
- diagnostic output.
The implementation should support both tagged and digest-pinned references.
Existing variables such as OPENSHELL_SUPERVISOR_IMAGE and OPENSHELL_DOCKER_SUPERVISOR_IMAGE must remain backward compatible. The implementation should either preserve them as aliases or define clear precedence between existing variables and the new common overrides.
A possible precedence order is:
- wrapper-specific existing override;
- common
SUPERVISOR_IMAGEorGATEWAY_IMAGEoverride; - registry-plus-tag fallback.
Alternatively, the project could use namespaced variables such as OPENSHELL_E2E_GATEWAY_IMAGE and OPENSHELL_E2E_SUPERVISOR_IMAGE to avoid collisions with unrelated tooling. If the simpler names are chosen, the documentation should clearly define their scope as test-wrapper inputs.
Acceptance Criteria
- All relevant OpenShell test wrappers support independent gateway and supervisor image selection.
- A custom gateway image can use a different repository, tag, or digest from the supervisor image.
- Existing workflows using only the registry prefix and image tag continue to resolve the same images as before.
- Kubernetes Helm deployments use the resolved gateway and supervisor image references.
- Local Kubernetes image builds and k3d/kind image loading use the resolved image references.
- Docker and Podman wrappers use the resolved image references consistently.
- Existing Docker and Podman supervisor override variables remain compatible.
- Full image tags and digest references are supported.
- Wrapper output identifies the resolved gateway and supervisor images.
-
TESTING.mddocuments the new overrides, fallback behavior, and precedence rules. - Tests or shell validation cover the default fallback and independent override cases.
Alternatives Considered
Continue requiring image retagging
Users could continue retagging custom images as ${REGISTRY_VALUE}/gateway:${IMAGE_TAG} and ${REGISTRY_VALUE}/supervisor:${IMAGE_TAG}. This avoids changing the wrappers but is inconvenient, obscures the images actually under test, and is difficult to use with immutable or externally managed images.
Add separate registry prefixes
Separate gateway and supervisor registry variables could be added while retaining a shared tag. This would help with different registries but would not support independently tagged or digest-pinned images.
Use additional Helm values files
For Kubernetes, users could provide Helm overrides for the gateway and supervisor images. This would not address Docker or Podman wrappers, local image builds, image imports, or generated gateway configuration.
Related Issues
- Issue #2067 discusses gateway and supervisor version compatibility, but does not address test-wrapper image overrides.
- Issue #1299 discusses Kubernetes supervisor image delivery, but focuses on image-volume sideloading.
- Issue #1597 discusses Kubernetes E2E harness behavior, but not custom gateway and supervisor image selection.
Agent Investigation
The current repository investigation found:
e2e/with-kube-gateway.shderives both images from a shared registry and tag in Helm installation, local builds, k3d imports, and kind image loading.- The Kubernetes wrapper also embeds the supervisor image when building the external Kubernetes driver gateway image.
e2e/with-docker-gateway.shsupportsOPENSHELL_DOCKER_SUPERVISOR_IMAGEandOPENSHELL_SUPERVISOR_IMAGE, but has no equivalent gateway-image override.e2e/with-podman-gateway.shsupportsOPENSHELL_SUPERVISOR_IMAGE, but does not provide a common gateway-image override.- Current CI workflows rely on the registry-plus-tag fallback, so preserving that fallback is required for backward compatibility.
- The repository documentation currently describes
OPENSHELL_REGISTRYandIMAGE_TAGas controlling the gateway/supervisor image pair.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading e2e/with-kube-gateway.sh, e2e/with-docker-gateway.sh, and e2e/with-podman-gateway.sh to trace how gateway and supervisor images flow through Helm, local builds, and image imports. Then inspect TESTING.md and existing supervisor override handling, preserving the registry-plus-tag fallback and defining precedence. Done means independent tagged or digest references work across the wrappers, diagnostics show the resolved images, and fallback and override cases are validated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, helm, kubernetes, shell
- Domain
- devops, documentation, infrastructure, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100