bug(supervisor): SPIFFE-enabled sandboxes crash because identity mount namespace is never prepared
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 8.7k
- Forks
- 1.3k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 253
Description
Agent Diagnostic
- Investigated a live Kubernetes deployment with OpenShell gateway and supervisor v0.0.92.
- Confirmed SPIRE server, agent, CSI driver, and OIDC provider were healthy before reproducing.
- Confirmed the sandbox Pod received the read-only
csi.spiffe.iovolume andOPENSHELL_PROVIDER_SPIFFE_WORKLOAD_API_SOCKET=/spiffe-workload-api/spire-agent.sock. - Confirmed SPIRE created an entry bound to the sandbox Pod UID with the expected per-sandbox SPIFFE ID and five-minute JWT-SVID TTL.
- Traced the failure to
crates/openshell-supervisor-process/src/process.rs:prepare_supervisor_identity_mount_namespace_from_env()is defined, whilesupervisor_identity_mount_from_env()requires itsOnceLock, but no caller exists. - Checked the latest published release, v0.0.92. The missing call is also still absent from the v0.0.95 source tag and current
mainat the time of filing. - Searched open and closed issues and pull requests. No open issue matches the exact failure. PR #2012 contains the exact diagnosis and fix but was automatically closed by the first-time-contributor vouch/DCO workflow before technical review. PR #2184 is related but addresses cleanup after
setns()failure, not the missing initializer call. - The upstream OpenShell agent skills were not exposed in the current harness; equivalent live-cluster, source, release, and duplicate diagnostics were performed manually.
Description
When provider_spiffe_workload_api_socket_path is configured for the Kubernetes driver, every sandbox supervisor crashes during startup with:
Failed to prepare supervisor identity isolation: supervisor identity mount namespace was not prepared before startup hardening
Expected behavior: the supervisor prepares the private mount namespace while privileged, retains access to the SPIFFE Workload API socket, hides that socket from sandbox child processes, and starts normally.
Actual behavior: the initializer is never called. The sandbox Pod enters a restart loop before a command can run, making SPIFFE-backed dynamic token grants unusable.
Reproduction Steps
- Deploy SPIRE and the SPIFFE CSI driver.
- Configure the Kubernetes driver with:
provider_spiffe_workload_api_socket_path = "/spiffe-workload-api/spire-agent.sock"
- Configure a
ClusterSPIFFEIDmatching gateway-managed sandbox Pods. - Create a sandbox using combined topology and the upstream v0.0.92 supervisor.
- Observe that:
- the Pod is scheduled and both images pull successfully;
- the CSI Workload API volume is mounted read-only;
- SPIRE creates the Pod-UID-bound registration entry;
- the supervisor container exits with code 1 and the error above.
- Inspect the tagged source and observe that
prepare_supervisor_identity_mount_namespace_from_env()has no caller.
The sandbox container had CAP_SYS_ADMIN, AppArmor: Unconfined, and valid node placement, so this is not the EPERM path addressed by #2184.
Environment
- OpenShell gateway:
0.0.92with a downstream gateway-only patch; supervisor is the unmodified upstreamghcr.io/nvidia/openshell/supervisor:0.0.92 - Local CLI used to initiate the API request:
0.0.69 - Kubernetes API: EKS
v1.36.2 - Node: Bottlerocket
1.62.1, Linux arm64, kubeletv1.36.0 - SPIRE chart:
0.29.0 - SPIFFE CSI driver:
0.2.7 - Latest published OpenShell release checked:
v0.0.92 - Newer source tag checked:
v0.0.95 - Existing issues and PRs checked: yes
Logs
Error: Failed to prepare supervisor identity isolation: supervisor identity mount namespace was not prepared before startup hardening
SPIRE successfully registered the same sandbox immediately before the supervisor failure:
Created entry ... spiffeID=spiffe://<trust-domain>/openshell/sandbox/<sandbox-uuid> ... selectors=[k8s:pod-uid:<pod-uid>] ... jwtSVIDTTL=5m0s
No credentials or token values were present in the sandbox specification or collected logs.
Likely Fix
PR #2012 proposes the minimal fix: call prepare_supervisor_identity_mount_namespace_from_env() in run_process() after privileged filesystem preparation and before startup hardening or child process creation. The fix should include a regression test proving both successful initialization and socket hiding from child processes.
Agent-First Checklist
- I pointed my agent at the repository and had it investigate this issue
- I loaded relevant upstream skills; they were unavailable in this harness, so equivalent diagnostics are documented above
- I checked the latest published OpenShell release and newer source tags
- I searched existing issues and pull requests for possible duplicates
- The issue could not be resolved without patching the upstream supervisor startup sequence
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in crates/openshell-supervisor-process/src/process.rs, especially run_process() and the existing prepare_supervisor_identity_mount_namespace_from_env() and supervisor_identity_mount_from_env() functions. Review PR #2012 and add a regression test covering successful initialization and socket hiding from child processes; the supervisor should start normally with the SPIFFE socket configured.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, rust
- Domain
- infrastructure, security
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100