Replace chart-inspection storage isolation with an annotation-driven, composed guardrail

Problem

KubePlus currently "enforces" storage isolation by checking that an application's Helm
chart does not itself define a StorageClass object with reclaimPolicy: Retain. This
check is close to vacuous: StorageClass is a cluster-scoped resource owned by platform
administrators, and an application chart — the thing being packaged as a KubePlus Kind —
would essentially never define one. The check passes by construction and doesn't actually
constrain what a tenant's application can do with storage.

There is no current mechanism limiting:

• which StorageClass a tenant's PersistentVolumeClaims may reference,


• how much storage a tenant can request, broken down by class,


• or preventing accidental use of an expensive/retain-policy class by a tenant who was
  never meant to have access to it.

Proposal

Move storage isolation from "inspect the chart's static templates" to "constrain the
rendered objects at admission time," the same way NetworkPolicy is already composed as a
sibling resource alongside a Kind instance.

Storage isolation properties cannot be added to the Kind instance's spec, because
spec on a KubePlus Kind instance directly represents the underlying Helm chart's
values.yaml — there is no values.yaml field for "which StorageClasses may this
instance use," and grafting one on would mean every chart author has to know about and
declare a field their chart never actually consumes. This needs to live outside spec,
the same way kubeplus.io/cross-ns-deps does for network dependencies: as a
KubePlus-owned annotation on the Kind instance CR, interpreted by KubePlus itself and
never passed through to Helm templating.

Two native Kubernetes mechanisms do the actual enforcement, composed from that annotation,
with no custom controller logic for the enforcement itself:

1. StorageClass-scoped ResourceQuota (caps how much).
Kubernetes ResourceQuota natively supports quota keys scoped to a specific StorageClass:
<storageclassname>.storageclass.storage.k8s.io/requests.storage and
<storageclassname>.storageclass.storage.k8s.io/persistentvolumeclaims. KubePlus already
composes a ResourceQuota per Kind instance for CPU/memory — extend it to include these
storage-class-scoped keys.

2. ValidatingAdmissionPolicy (caps which classes are allowed at all).
ValidatingAdmissionPolicy is a built-in Kubernetes API (no OPA/Kyverno dependency) that
can reject a PersistentVolumeClaim referencing a disallowed StorageClass, expressed in
CEL, scoped to the tenant's namespace via a ValidatingAdmissionPolicyBinding.

Annotation name and shape

Annotation key: kubeplus.io/storage-isolation

Placed on the KubePlus Kind instance CR, alongside kubeplus.io/cross-ns-deps if both
apply to the same instance. Value is a single JSON object (unlike cross-ns-deps, this
isn't naturally a list — an instance has one storage policy, not several):

apiVersion: cloudark.io/v1

kind: Agent

metadata:

  name: team-a-agent-instance

  namespace: team-a

  annotations:

    kubeplus.io/storage-isolation: |

      {

        "allowedClasses": ["standard-rwo"],

        "maxRequestGB": 50,

        "maxClaims": 5

      }

spec:

  mcpServer:

    name: shared-k8sgpt-mcp

    namespace: platform-mcp

  modelConfigRef: default-model-config

Field | Required | Meaning
-- | -- | --
allowedClasses | no | Allow-list of StorageClass names the instance's PVCs may use. Omit to allow any class already permitted at the namespace/cluster level (no additional restriction).
maxRequestGB | no | Cap on total requests.storage, scoped per class if allowedClasses has exactly one entry, else applied as an aggregate across all listed classes.
maxClaims | no | Cap on number of PVCs, scoped the same way as maxRequestGB.

How KubePlus should handle the annotation

On Kind instance create/update:

1. Watch for kubeplus.io/storage-isolation on any KubePlus Kind instance, across all
   registered Kinds (Kind-agnostic, same as kubeplus.io/cross-ns-deps — KubePlus never
   needs to know what a given Kind's chart does with storage).


2. Parse the JSON object. If the annotation is absent, generate no storage-specific
   objects (backward compatible with instances that don't declare this).


3. For each class in allowedClasses (or a single unscoped block if allowedClasses is
   omitted but maxRequestGB/maxClaims are set): add or update the corresponding keys
   on the instance's existing per-instance ResourceQuota object. This is a merge into
   the quota object KubePlus already manages for the instance, not a new object.


4. If allowedClasses is set, generate a deterministically-named
   ValidatingAdmissionPolicy + ValidatingAdmissionPolicyBinding pair scoped to the
   instance's namespace: kubeplus-storageclass-allowlist-<instance-name>, rejecting any
   PersistentVolumeClaim in that namespace whose spec.storageClassName is not in the
   allow-list.


5. Reconcile on any change to the annotation value (e.g. a class added to or removed from
   allowedClasses, or the caps changed).

On Kind instance delete:

1. Remove the storage-scoped keys from the instance's ResourceQuota (or delete the
   quota object entirely if KubePlus already does that on instance deletion today).


2. Delete the ValidatingAdmissionPolicy/ValidatingAdmissionPolicyBinding pair
   associated with this instance. This object is per-instance, not shared across
   consumers the way the cross-namespace-deps ingress NetworkPolicy is, so it's always
   safe to delete unconditionally on instance deletion — no reference-counting needed.

Acceptance criteria

• [ ] An instance with no kubeplus.io/storage-isolation annotation behaves exactly as
  today (no behavior change, no new objects).


• [ ] An instance whose annotation declares allowedClasses rejects a
  PersistentVolumeClaim in its namespace that references a class outside the list,
  at admission time (not reconciliation time — the tenant gets immediate kubectl apply feedback).


• [ ] An instance whose annotation declares maxRequestGB is blocked by quota, not by
  the admission policy, once its PVCs collectively exceed the cap.


• [ ] Editing the annotation on a live instance (e.g. adding a class to allowedClasses)
  updates the existing ValidatingAdmissionPolicyBinding and quota keys in place,
  without requiring the instance to be recreated.


• [ ] Deleting the instance removes both the quota keys and the admission policy
  objects; no dangling ValidatingAdmissionPolicyBinding remains.

Demo steps

1. Instantiate an Agent Kind instance with:
   

   metadata:  annotations:    kubeplus.io/storage-isolation: |      {"allowedClasses": ["standard-rwo"], "maxRequestGB": 10}
   
   



2. Attempt to create a PVC in that namespace referencing a different StorageClass (e.g.
   one with reclaimPolicy: Retain) — show the admission rejection:
   

   kubectl apply -f pvc-wrong-class.yaml   # expect: denied by ValidatingAdmissionPolicy
   
   



3. Create a PVC referencing standard-rwo for 15Gi — show the quota rejection:
   

   kubectl apply -f pvc-too-big.yaml       # expect: denied by ResourceQuota
   
   



4. Create a compliant PVC (standard-rwo, 5Gi) — show it succeeds.


5. Edit the instance's annotation to add a second allowed class, and show the
   ValidatingAdmissionPolicyBinding updates without recreating the instance:
   

   kubectl annotate agent team-a-agent-instance -n team-a --overwrite \  kubeplus.io/storage-isolation='{"allowedClasses":["standard-rwo","fast-ssd"],"maxRequestGB":10}'kubectl get validatingadmissionpolicybinding -n team-a -o yaml
   
   



6. Delete the instance, then show the ValidatingAdmissionPolicyBinding and quota keys
   are gone:
   

   kubectl get validatingadmissionpolicybinding -n team-akubectl get resourcequota -n team-a -o yaml
   
   

No files or tests are named. Start by locating the existing per-instance ResourceQuota reconciliation and the NetworkPolicy composition used for cross-namespace dependencies, then trace how Kind instance create, update, and delete events are handled. Done means the storage-isolation annotation drives quota keys and admission policy objects, while unannotated instances remain unchanged and deletion leaves no dangling objects.

Written by the indexing model from the issue text.