NVIDIA / NVIDIA/OpenShell

feat(server): support more than one sandbox ServiceAccount per gateway

Open
#2,806 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area:compute area:gateway state:accepted test:e2e-kubernetes
Dominant language
Rust
Stars
8.7k
Forks
1.3k
Avg merge
2d 11h
Merged PRs (30d)
253

Description

Description

The sandbox ServiceAccount is fixed gateway-wide. [openshell.drivers.kubernetes] service_account_name is a single value that serves two purposes at once:

  1. the driver stamps it onto every sandbox pod as serviceAccountName
    https://github.com/NVIDIA/OpenShell/blob/8d67250a5d17348eb96c4fa46226b06d8041f2ba/crates/openshell-driver-kubernetes/src/driver.rs#L3517
  2. bootstrap auth requires the presented principal to match it exactly —
    https://github.com/NVIDIA/OpenShell/blob/8d67250a5d17348eb96c4fa46226b06d8041f2ba/crates/openshell-server/src/auth/k8s_sa.rs#L386

So every sandbox on a gateway necessarily shares one Kubernetes identity. This asks for that to be configurable as a set, in the same spirit as the namespace-level validation added in NVIDIA/OpenShell#2656.

Context

The ServiceAccount is the unit of cloud IAM federation. On EKS the IAM trust policy conditions on the token subject:

StringLike  <oidc>:sub = "system:serviceaccount:<namespace>:<service-account>"

GCP Workload Identity and Azure Workload Identity bind the same way. So "one SA per gateway" means "one cloud identity per gateway": sandboxes doing different work, with different blast radii, cannot be granted different cloud permissions. Granting the union to the shared SA gives every sandbox the most-privileged set.

Notably the SA is the only pod-level field with no per-sandbox override. platform_config on the SandboxTemplate already carries runtime_class_name, node_selector, tolerations, annotations and host_users, including a documented per-sandbox-overrides-cluster-default precedent:

https://github.com/NVIDIA/OpenShell/blob/8d67250a5d17348eb96c4fa46226b06d8041f2ba/crates/openshell-driver-kubernetes/src/driver.rs#L3503

Workarounds today, both unattractive:

  1. Share one SA and grant it the union of permissions — every sandbox gets every permission.
  2. Run a gateway per identity — preserves isolation, but duplicates the gateway, its database and its forwarder topology for what is a single config value.

Proposed Direction

Two complementary parts; either is useful, together they close the case:

  1. Accept a configured set of sandbox service accounts in bootstrap auth. A single value stays the default and keeps current behaviour.
  2. Allow the pod's ServiceAccount to be selected per sandbox — e.g. platform_config.service_account_name, constrained to the set from (1), defaulting to the existing single value.

They are separable. (2) without (1) cannot authenticate, so (1) is the prerequisite. (1) alone is already useful for deployments where something other than the gateway sets the pod's ServiceAccount — a mutating admission policy, or an external controller that owns the sandbox pods — which is the situation we hit: our pods are given a different SA out-of-band, and bootstrap then fails with

K8s TokenReview principal is not the configured sandbox service account

Whichever way (1) is implemented, it should be additive rather than a widening of service_account_name, because of consumer (1) in the Description: turning that field into a list or a delimited string would render invalid pods. The pod default and the accepted-for-auth set need to be distinct settings.

This overlaps NVIDIA/OpenShell#2023, which proposes making bootstrap config explicit and independent of the selected compute driver, and whose first Definition-of-Done item is defining the ownership model for the bootstrap namespace and service-account settings. That seems like the natural place to decide where an accepted set lives; happy to follow that issue's outcome rather than pre-empt it.

Security properties are unchanged: tokens are still verified through TokenReview, still required to be pod-bound (pod_name / pod_uid extras), and IssueSandboxToken still requires a K8sServiceAccount-sourced sandbox principal. This widens which identities an operator may enrol, not how they are verified — the accepted set stays operator-configured and closed.

Definition of Done

  • Bootstrap auth accepts a configured set of sandbox service accounts; a single value remains the default and is unchanged in behaviour.
  • The pod-default service account and the accepted-for-auth set are distinct settings, so the driver keeps stamping exactly one name onto pods.
  • Rejection logging identifies the presented principal and that it was not in the accepted set.
  • Tests: single configured SA (accept + reject), multiple configured SAs (each accepted, non-member rejected).
  • Helm chart can express the set, with the existing single-value setting still supported.
  • Gateway configuration documentation updated.

Environment

Observed on gateway 0.0.85; code references above are against main at 8d67250a.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with crates/openshell-driver-kubernetes/src/driver.rs and crates/openshell-server/src/auth/k8s_sa.rs at the referenced entry points, then review the namespace validation from NVIDIA/OpenShell#2656 and the ownership discussion in #2023. Done means distinct pod-default and accepted-service-account settings, single- and multi-account acceptance tests, rejection logging, Helm support, and updated gateway configuration documentation.

Written by the indexing model from the issue text.

Assessment

Tech stack
helm, kubernetes, rust
Domain
authentication, cloud, infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.