GPU isolation via a namespace-scoped mutating webhook and composed DRA objects

Problem

KubePlus has no notion of GPU isolation today. GPU access is a real multi-tenancy concern
for any Kind instance with a GPU-backed component anywhere in its execution graph — not
only when the agent's primary model is served locally. Even an agent whose main reasoning
loop calls an external LLM API can depend on GPU-backed components elsewhere: an
embedding or reranking model behind a RAG tool, a local classifier used for guardrails or
PII detection, a vision/OCR/speech tool, or a small local model used for cheap
intermediate steps before escalating to an external API. Whether the top-level LLM call
goes to an external provider is largely orthogonal to whether GPU isolation matters — what
matters is whether any component the instance depends on, directly or via a shared MCP
server, touches a GPU.

Kubernetes' GPU story has shifted meaningfully in the last few months: Dynamic Resource
Allocation (DRA) graduated to GA in Kubernetes 1.34 and is on by default from there,
replacing the old opaque-integer device-plugin model with structured objects
(DeviceClass, ResourceClaim, ResourceClaimTemplate, ResourceSlice). NVIDIA has also
donated its DRA driver for GPUs to the CNCF, moving it toward community ownership rather
than a single-vendor plugin.

Referencing a DRA claim from a Pod requires two things in the Pod spec: a pod-level
spec.resourceClaims entry naming the claim source, and a per-container
resources.claims entry referencing that name. Requiring a third-party chart's
values.yaml to expose fields for either of these is the same unrealistic assumption
flagged in the node-isolation issue — a chart not written with KubePlus in mind has no
reason to expose a resourceClaims field, and there's no equivalent of a
StorageClass-style cluster default to fall back on.

Three GPU sharing strengths exist, with different isolation/utilization trade-offs:

Mechanism | Isolation | Notes
-- | -- | --
Full/exclusive GPU | Strongest (whole device) | Lowest utilization for bursty agent workloads
MIG (Multi-Instance GPU) | Hardware-partitioned, up to 7 instances on A100/H100/H200/B200 | Fixed-size partitions; supported on Ampere-generation+ GPUs only
Time-slicing | None — no memory/fault isolation | High density, unsafe for mutually untrusting tenants
MPS | Soft memory limits only | Better utilization than time-slicing, still not hard isolation

When KubePlus processes this annotation, it writes the resolved claim-template name onto
the Namespace object:

apiVersion: v1

kind: Namespace

metadata:

  name: team-a

  labels:

    kubeplus.io/gpu-isolation-enabled: "true"

  annotations:

    kubeplus.io/gpu-isolation-resolved: |

      {"claimTemplateName": "kubeplus-gpu-team-a-agent-instance"}

As with node isolation, the label exists purely so namespaceSelector can match on it
cheaply; the annotation carries the only value the webhook actually needs, so the webhook
never has to read the instance CR or the ResourceClaimTemplate object at request time —
only the Namespace, via its own informer cache.

Mutating webhook configuration changes

A separate webhooks[] entry, independent of both the existing CRD-focused rule and the
node-isolation Pod rule (multiple mutating webhook entries on Pods compose fine, since
each only patches the specific paths it owns):

{

  "name": "gpu-isolation.kubeplus.io",

  "rules": [

    {

      "apiGroups": [""],

      "apiVersions": ["v1"],

      "operations": ["CREATE"],

      "resources": ["pods"],

      "scope": "Namespaced"

    }

  ],

  "namespaceSelector": {

    "matchExpressions": [

      {

        "key": "kubeplus.io/gpu-isolation-enabled",

        "operator": "In",

        "values": ["true"]

      }

    ]

  },

  "failurePolicy": "Ignore",

  "sideEffects": "None",

  "admissionReviewVersions": ["v1"]

}

failurePolicy: Ignore for the same reason as node isolation: brief webhook
unavailability should fail open rather than blocking all pod creation in the cluster.
Unlike node isolation, though, failing open here means a pod could start without its GPU
claim at all — worth flagging as a real trade-off rather than a purely cosmetic one, since
"pod runs but can't see a GPU" is a more disruptive failure mode than "pod runs on any
node." If that trade-off isn't acceptable, failurePolicy: Fail is the alternative, at
the cost of blocking pod admission in GPU-isolated namespaces during webhook downtime.

JSON patch implementation (Go)

package gpuisolation


import (

	"encoding/json"

	"fmt"

	corev1 "k8s.io/api/core/v1"

)

// GPUIsolationSpec is the resolved value copied onto a Namespace's

// kubeplus.io/gpu-isolation-resolved annotation.

type GPUIsolationSpec struct {

	ClaimTemplateName string `json:"claimTemplateName"`

}

const claimRefName = "kubeplus-gpu"

// namespaceCache is populated by the same Namespace informer used for node

// isolation and kept current via Add/Update/Delete event handlers. Lookups

// are in-memory map reads keyed by namespace name — no API calls happen per

// admission request.

type namespaceCache interface {

	Get(namespace string) (*GPUIsolationSpec, bool)

}

// BuildGPUIsolationPatch returns a JSON Patch (RFC 6902) that adds a

// pod-level resourceClaims entry pointing at the instance's

// ResourceClaimTemplate, and references that claim from every container's

// resources.claims list. Existing pod-level resourceClaims and per-container

// resources.claims entries, if any, are preserved rather than clobbered.

func BuildGPUIsolationPatch(pod *corev1.Pod, iso *GPUIsolationSpec) ([]byte, error) {

	var patches []map[string]interface{}

	claimEntry := map[string]interface{}{

		"name": claimRefName,

		"source": map[string]interface{}{

			"resourceClaimTemplateName": iso.ClaimTemplateName,

		},

	}

	if len(pod.Spec.ResourceClaims) == 0 {

		patches = append(patches, map[string]interface{}{

			"op":    "add",

			"path":  "/spec/resourceClaims",

			"value": []interface{}{claimEntry},

		})

	} else {

		patches = append(patches, map[string]interface{}{

			"op":    "add",

			"path":  "/spec/resourceClaims/-",

			"value": claimEntry,

		})

	}

	claimRef := map[string]interface{}{"name": claimRefName}

	for i, c := range pod.Spec.Containers {

		if len(c.Resources.Claims) == 0 {

			patches = append(patches, map[string]interface{}{

				"op":    "add",

				"path":  fmt.Sprintf("/spec/containers/%d/resources/claims", i),

				"value": []interface{}{claimRef},

			})

		} else {

			patches = append(patches, map[string]interface{}{

				"op":    "add",

				"path":  fmt.Sprintf("/spec/containers/%d/resources/claims/-", i),

				"value": claimRef,

			})

		}

	}

	return json.Marshal(patches)

}

// HandleAdmission is the webhook's per-request entry point. Given the

// existing namespaceSelector-based filtering at the MutatingWebhookConfiguration

// level, every request reaching here is already known to be a Pod CREATE in a

// namespace labeled kubeplus.io/gpu-isolation-enabled=true.

func HandleAdmission(namespace string, pod *corev1.Pod, cache namespaceCache) (patch []byte, err error) {

	iso, ok := cache.Get(namespace)

	if !ok {

		// Label present but annotation missing/unparseable — fail open,

		// consistent with failurePolicy: Ignore at the config level.

		return nil, nil

	}

	return BuildGPUIsolationPatch(pod, iso)

}

Known limitation, called out deliberately rather than hidden: the patch above adds
the claim reference to every container in the pod, including sidecars that may not
actually touch the GPU. This is a conscious simplification — identifying which specific
container needs the device would require chart-specific knowledge (a container name
convention, an annotation, etc.) that reintroduces exactly the kind of per-chart
assumption this design is trying to avoid. Referencing an unused claim from a sidecar is
inert (it grants visibility, not automatic usage), so this is a reasonable default; revisit
only if a concrete chart surfaces a problem with it.

Enforcing maxDevices via ResourceQuota

Kubernetes supports quota scoped to DRA claim counts natively:
count/resourceclaims.resource.k8s.io. KubePlus adds this key to the instance's existing
per-instance ResourceQuota object (the same object storage isolation already extends),
set from maxDevices — no admission-policy or webhook logic needed for this part, since
ResourceQuota enforcement is built into the API server.

Acceptance criteria

• [ ] An instance with no kubeplus.io/gpu-isolation annotation gets no GPU device claim
  and no Pod mutation (default-deny).


• [ ] sharing: mig produces a ResourceClaimTemplate/DeviceClass pair; two instances
  requesting the same migProfile on the same node get isolated, non-overlapping MIG
  instances.


• [ ] A Pod created in a GPU-isolated namespace receives the pod-level resourceClaims
  entry and per-container resources.claims reference without the chart itself
  having declared either field, and without any pre-existing resourceClaims/
  resources.claims entries the chart did declare being removed.


• [ ] Webhook latency for Pod admission in a GPU-isolated namespace does not meaningfully
  increase with chart complexity, for the same reason as node isolation: no ownership
  traversal is performed.


• [ ] Exceeding maxDevices is rejected by ResourceQuota, not the webhook or an
  admission policy.


• [ ] Deleting the instance removes the ResourceClaimTemplate, the quota keys, and the
  namespace label/annotation naturally via namespace deletion, with no explicit
  cleanup step required.

Demo steps

1. On a node with MIG enabled (or simulated via a MIG-capable GPU in the cluster),
   instantiate an Agent Kind instance with:
   

   metadata:  annotations:    kubeplus.io/gpu-isolation: |      {"sharing": "mig", "migProfile": "1g.10gb", "maxDevices": 1}
   
   



2. Confirm the namespace carries the label and resolved annotation, and that the
   ResourceClaimTemplate exists:
   

   kubectl get namespace team-a -o jsonpath='{.metadata.labels}{"\n"}{.metadata.annotations}'kubectl get resourceclaimtemplate -n team-a
   
   



3. Confirm a pod in that namespace picks up the claim without the chart declaring it:
   

   kubectl get pod <pod-name> -n team-a -o jsonpath='{.spec.resourceClaims}'kubectl get pod <pod-name> -n team-a -o jsonpath='{.spec.containers[*].resources.claims}'
   
   



4. Attempt to create a second GPU-requesting pod in the same namespace beyond
   maxDevices: 1 and show the ResourceQuota rejection.


5. Delete the instance and confirm the ResourceClaimTemplate and quota keys are gone:
   

   kubectl delete agent team-a-agent-instance -n team-akubectl get resourceclaimtemplate -n team-a   # expect: not found
   
   

Research note: hybrid MIG + time-slicing for agent workloads

hybrid mode is deliberately included as an open research question rather than a fully
specified mechanism: use MIG to give each tenant a hardware-isolated partition as a
baseline guarantee, then apply time-slicing within that partition across that same
tenant's own bursty invocations to improve utilization, without extending time-slicing's
lack of isolation across tenant boundaries. Whether this is worth the added scheduling
complexity for agent-specific traffic patterns versus just provisioning a slightly larger
MIG partition per tenant is an open evaluation question. Treat hybrid as experimental
until there's utilization data comparing it against mig-only and exclusive under
realistic agent traffic.

Start by locating the existing node-isolation webhook and Namespace informer, then trace the ResourceQuota reconciliation for instance storage isolation. Use the BuildGPUIsolationPatch and HandleAdmission entry points as the implementation outline, and verify the kubectl demo steps. Done means DRA claims are composed without clobbering existing fields, maxDevices is quota-enforced, and the acceptance criteria pass.

Written by the indexing model from the issue text.