Layered isolation: an annotation governing which Agent Sandbox runtime a tenant may use

Problem / motivation

kubernetes-sigs/agent-sandbox (moving toward a CNCF project) provides a Sandbox CRD
(agents.x-k8s.io/v1alpha1) for declaratively managing long-running, stateful, singleton
execution environments for a single agent's untrusted code execution or computer use —
with stable identity, pause/resume/shutdown lifecycle, and warm pools for fast cold start,
isolated via gVisor or Kata Containers through the standard runtimeClassName
mechanism.

This is a different isolation axis than anything KubePlus currently manages. KubePlus's
mechanisms isolate tenant from tenant — namespace, quota, network, storage, node
placement. Agent Sandbox isolates the host from a single untrusted execution — one
agent's code-interpreter or shell tool call, running inside one Sandbox object.

Left alone, these two layers don't compose automatically: a tenant whose namespace
KubePlus has carefully isolated (NetworkPolicy, quota, dedicated node pool) could still
create Sandbox objects with runtimeClassName: runc (no sandboxing at all), or with no
runtimeClassName set, silently bypassing the isolation-backend guarantees the platform
team intended for that tenant. There's currently no mechanism tying "how isolated is this
tenant's environment" to "how isolated is the code this tenant's agent actually executes."

Proposal

Don't have KubePlus reimplement or wrap Agent Sandbox's execution-isolation logic — Agent
Sandbox already solves that well, and duplicating it would mean KubePlus taking on
Kind-specific logic about a schema it doesn't own (exactly the problem the
kubeplus.io/cross-ns-deps annotation design was built to avoid). Instead, add a second,
fully independent KubePlus-owned annotation governing sandbox-runtime policy, and enforce
it — via an admission policy, not a webhook — against any Sandbox object created in that
tenant's namespace.

Note on scope: kubeplus.io/sandbox-isolation is fully independent of
kubeplus.io/node-isolation. It always generates its own, standalone RuntimeClass
object, unconditionally, regardless of whether kubeplus.io/node-isolation is present on
the same instance. The two annotations are orthogonal — node isolation governs Pod
placement (see the node-isolation issue), while sandbox isolation composes a
RuntimeClass purely to carry an isolation-backend handler for Sandbox objects to
reference. There is no shared object between them, and no coordination is required
between the two.

This keeps the division of responsibility clean:

• KubePlus governs the "who" and "how strong": which tenant, and what the minimum
  required isolation backend is for anything running in that tenant's namespace.


• Agent Sandbox governs the "what happens inside one execution": lifecycle, identity,
  warm pools, the actual gVisor/Kata mechanics.

Annotation name and shape

Annotation key: kubeplus.io/sandbox-isolation

apiVersion: cloudark.io/v1

kind: Agent

metadata:

  name: team-a-agent-instance

  namespace: team-a

  annotations:

    kubeplus.io/node-isolation: |

      {"nodeSelector": {"pool": "team-a-pool"}, "tolerations": [...]}

    kubeplus.io/sandbox-isolation: |

      {

        "minRuntimeClass": "kata",

        "enforce": true

      }

spec:

  mcpServer:

    name: shared-k8sgpt-mcp

    namespace: platform-mcp

  modelConfigRef: default-model-config

Both annotations may appear on the same instance, as shown above, but they're handled by
entirely independent reconciliation paths — presence or absence of one has no bearing on
the other's behavior.

Field | Required | Meaning
-- | -- | --
minRuntimeClass | yes | Minimum isolation-backend handler required for any Sandbox object created in this instance's namespace. Ordered runc < runsc < kata for the purpose of the "minimum" comparison.
enforce | no, default true | If true, KubePlus generates an admission policy that rejects non-conforming Sandbox objects. If false, KubePlus only sets a namespace default (see below) without hard-blocking exceptions — useful for staged rollout.

How KubePlus should handle the annotation

On Kind instance create/update:

1. Watch for kubeplus.io/sandbox-isolation on any KubePlus Kind instance
   (Kind-agnostic — KubePlus never needs to understand the Sandbox CRD's own schema
   beyond its runtimeClassName field).


2. Generate a deterministically-named RuntimeClass object,
   kubeplus-sandbox-<instance-name>, with only handler set (runsc for gVisor,
   kata for Kata) — no scheduling fields. This happens unconditionally whenever the
   annotation is present, independent of kubeplus.io/node-isolation.


3. Generate a deterministically-named ValidatingAdmissionPolicy +
   ValidatingAdmissionPolicyBinding scoped to the instance's namespace, targeting the
   agents.x-k8s.io/v1alpha1 Sandbox resource: reject any Sandbox whose
   runtimeClassName resolves to a handler weaker than minRuntimeClass, or whose
   runtimeClassName is unset, when enforce: true.


4. If enforce: false, instead set a namespace-level default so unset
   runtimeClassName fields are populated automatically (via the cluster's default
   RuntimeClass mechanism, if configured) without rejecting explicit overrides — this is
   the "advisory" mode for teams not ready for hard enforcement.


5. Reconcile on annotation changes; tightening minRuntimeClass does not retroactively
   affect already-running Sandboxes, only new ones — document this as expected behavior.

On Kind instance delete:

1. Delete the ValidatingAdmissionPolicy/ValidatingAdmissionPolicyBinding pair — safe
   unconditionally, per-instance.


2. Delete the kubeplus-sandbox-<instance-name> RuntimeClass object — also safe
   unconditionally, since it is always instance-owned.


3. Existing Sandbox objects in the namespace are not deleted by KubePlus — they're
   owned by whatever created them (the agent's own controller, or a human), and Agent
   Sandbox's own lifecycle/shutdown mechanisms apply.

Acceptance criteria

• [ ] An instance with no kubeplus.io/sandbox-isolation annotation behaves exactly as
  today — no constraint on Sandbox objects in its namespace, and no RuntimeClass
  generated on its behalf.


• [ ] An instance with minRuntimeClass: kata, enforce: true rejects a Sandbox
  object created in its namespace with runtimeClassName: runc or unset, at
  admission time.


• [ ] The same instance accepts a Sandbox object with runtimeClassName resolving to
  the kubeplus-sandbox-<instance-name> RuntimeClass KubePlus generated for it.


• [ ] An instance carrying both kubeplus.io/node-isolation and
  kubeplus.io/sandbox-isolation produces exactly one RuntimeClass object overall
  — kubeplus-sandbox-<instance-name>.


• [ ] enforce: false populates the default runtimeClassName for unset Sandboxes
  without rejecting an explicit weaker choice (advisory-only behavior verified).


• [ ] Deleting the Kind instance removes both the admission policy objects and the
  RuntimeClass; pre-existing Sandbox objects in the namespace are untouched.

Demo steps

1. Install the Agent Sandbox controller and CRDs in the cluster alongside kagent and
   KubePlus (continuing the running example: shared-k8sgpt-mcp in platform-mcp,
   agent instances in team-a/team-b).


2. Instantiate team-a-agent-instance with both annotations:
   

   metadata:  annotations:    kubeplus.io/node-isolation: |      {"nodeSelector": {"pool": "team-a-pool"}, "tolerations": [...]}    kubeplus.io/sandbox-isolation: |      {"minRuntimeClass": "kata", "enforce": true}
   
   

   Confirm exactly one RuntimeClass object exists in the cluster for this instance —
   kubeplus-sandbox-team-a-agent-instance, with handler: kata:
   

   kubectl get runtimeclass -o name | grep team-a-agent-instance
   
   



3. Confirm node placement is also in effect, via the mandated nodeSelector on the
   instance's pods:
   

   kubectl get pods -n team-a -o jsonpath='{.items[*].spec.nodeSelector}'
   
   



4. Attempt to create a Sandbox in team-a without a runtimeClassName — show the
   admission rejection:
   

   kubectl apply -f sandbox-no-runtimeclass.yaml   # expect: denied
   
   



5. Create a Sandbox in team-a referencing
   kubeplus-sandbox-team-a-agent-instance — show it's accepted and starts:
   

   kubectl apply -f sandbox-kata.yamlkubectl get sandbox -n team-a
   
   



6. Instantiate team-b-agent-instance with only kubeplus.io/sandbox-isolation
   (enforce: false, no node-isolation annotation at all), create a Sandbox with no
   runtimeClassName, and show it gets the namespace default applied rather than being
   rejected — contrasting enforced vs. advisory mode, and confirming sandbox-isolation
   works identically whether or not node-isolation is present on the instance.


7. Wrap-up talking point: this is the same "KubePlus composes a governed guardrail from a
   declarative annotation, without needing to understand the dependent Kind's schema"
   pattern used for cross-namespace MCP access, storage, and node isolation — applied
   here to govern a brand-new, fast-moving execution-sandboxing standard without KubePlus
   coupling to its internals.

Start by tracing KubePlus's existing annotation reconciliation and node-isolation path, then examine how it creates RuntimeClass and admission-policy resources. Use the listed kubectl demo commands to verify enforced and advisory Sandbox behavior, independent node isolation, and cleanup on instance deletion.

Written by the indexing model from the issue text.