NVIDIA / NVIDIA/OpenShell

bug: Kubernetes user namespace E2E uses incompatible kubectl setup

Open
#1,597 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area:cluster state:stale test:e2e-kubernetes topic:testing
Dominant language
Rust
Stars
8.7k
Forks
1.3k
Avg merge
2d 11h
Merged PRs (30d)
253

Description

Agent Diagnostic

Loaded skill: create-github-issue.

Investigation findings:

  • The user_namespaces Rust E2E target is declared in e2e/rust/Cargo.toml with required-features = ["e2e-kubernetes"].
  • On main, mise run e2e:kubernetes did not normally execute this target because e2e/rust/e2e-kubernetes.sh defaulted to e2e,e2e-host-gateway, without e2e-kubernetes.
  • When the target is selected, the test setup is incompatible with the repo-controlled Kubernetes E2E flow. e2e/rust/tests/user_namespaces.rs invokes kubectl as docker exec openshell-cluster-openshell kubectl, but e2e/with-kube-gateway.sh, the CI kind workflow, and the local k3d flow use host kubectl --context ...; they do not create a Docker container named openshell-cluster-openshell.
  • Even after replacing that hardcoded docker exec, the test mutates the gateway StatefulSet with kubectl set env, which triggers a gateway rollout while with-kube-gateway.sh is holding kubectl port-forward sessions to the gateway service/statefulset. That can break the CLI endpoint used by openshell sandbox create and surface as a generic sandbox creation timeout.

Relevant files:

  • e2e/rust/tests/user_namespaces.rs
  • e2e/rust/e2e-kubernetes.sh
  • e2e/with-kube-gateway.sh
  • .github/workflows/e2e-kubernetes-test.yml

Description

Actual behavior: When the Kubernetes user namespace E2E test is selected, it is likely to fail before exercising user namespace behavior because it shells out through a hardcoded Docker container name that the standard CI/local Kubernetes setup does not create. If that is fixed directly, the test can still be flaky or fail because it restarts the gateway in the middle of the run and disrupts active port-forwards.

Expected behavior: The user namespace E2E test should run against the same Kubernetes setup as the rest of mise run e2e:kubernetes, using the configured kube context and installing the gateway with user namespaces enabled before the test command starts. It should not perform a mid-test gateway rollout.

Reproduction Steps

  1. Ensure the user_namespaces target is selected by running Kubernetes E2E with the e2e-kubernetes feature, or by targeting the test directly.
  2. Run the Kubernetes E2E path against the repo-managed kind/k3d setup.
  3. Observe that the test's kubectl helper attempts docker exec openshell-cluster-openshell kubectl ... instead of using the configured kube context.
  4. If the kubectl helper is patched ad hoc, observe that kubectl set env statefulset/openshell performs a gateway rollout during the test and can break the wrapper's port-forwarded gateway endpoint.

Environment

  • OS: Linux CI / local Kubernetes E2E hosts
  • Kubernetes E2E setup: e2e/with-kube-gateway.sh using kind in CI or k3d/existing context locally
  • OpenShell: current main and branches that select e2e-kubernetes by default

Logs

Expected failure mode when selected under the standard setup:

kubectl [..] failed: Error response from daemon: No such container: openshell-cluster-openshell

Potential follow-on failure mode after replacing the hardcoded Docker exec:

sandbox <name> did not appear within 60s

Suggested Fix

  • Replace the test-local docker exec openshell-cluster-openshell kubectl helper with a shared E2E helper that uses the same kube context as with-kube-gateway.sh.
  • Add a wrapper-level way to install the Helm chart with server.enableUserNamespaces=true before tests start, for example an OPENSHELL_E2E_KUBE_ENABLE_USER_NAMESPACES=1 switch that appends the appropriate Helm --set.
  • Remove the mid-test kubectl set env / gateway rollout from user_namespaces.rs.
  • Keep the test focused on creating a sandbox, waiting for the Sandbox CR/pod, inspecting spec.hostUsers=false and the expected capabilities, and cleaning up.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with e2e/rust/tests/user_namespaces.rs and compare its kubectl and gateway setup with e2e/rust/e2e-kubernetes.sh, e2e/with-kube-gateway.sh, and .github/workflows/e2e-kubernetes-test.yml. Run the Kubernetes E2E path with the repo-managed kind or k3d setup; done means the test uses the configured kube context, enables user namespaces before startup, avoids a mid-test rollout, and verifies the sandbox behavior successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, helm, kubernetes, rust
Domain
devops, infrastructure, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.