Dedicated-mode node isolation via a narrowly-scoped Pod mutating webhook rule

### Problem

KubePlus previously enforced node isolation (pinning a Kind instance's pods to specific
nodes) via a mutating webhook that had to determine instance ownership by traversing
Pod → ReplicaSet → Deployment → custom resource for every Pod, synchronously, inside the
admission request. For charts containing custom resources this traversal was expensive
enough to risk the webhook's ~30s timeout, and it made instance creation slow. That's why
node isolation was dropped in favor of asking chart authors to externalize
`nodeName`/`nodeSelector` in `values.yaml` — which works, but gives KubePlus no place to
actually *enforce* placement, and is an unreasonable assumption for third-party charts
that were never written with KubePlus in mind.

This issue is scoped to **dedicated mode only** — hard placement onto a tenant-specific,
typically tainted node pool. (Soft, preference-only placement without exclusivity is a
separate, already-simpler mechanism — `PodNodeSelector` — and isn't covered here.)

### Proposal

The expensive part of the old design was determining *which instance a Pod belongs to*.
That problem disappears entirely once ownership is reduced to *which namespace a Pod is
in* — true here because KubePlus already creates one namespace per instance. With that,
the webhook needs no traversal and no API calls per request at all:

1. **Scope webhook invocation itself, cheaply, via `namespaceSelector`.** The API server
evaluates `namespaceSelector` before ever calling the webhook — so pods in namespaces
without the isolation label never reach KubePlus's webhook code, cluster-wide, at zero
cost to KubePlus.
2. **Resolve the actual nodeSelector/tolerations from an in-memory cache, not a live
query.** KubePlus's controller writes the resolved values onto the Namespace object
itself (as an annotation) when it processes the instance's annotation. The webhook
process keeps a Namespace informer in memory; resolving a request is a map lookup
keyed by `request.Namespace`, not an API call.
3. **The patch itself only ever touches two fixed, top-level `PodSpec` fields** —
`nodeSelector` and `tolerations` — whose location and shape never vary regardless of
how deep or CR-heavy the chart's object graph is.

Together, per-request webhook latency is now constant-time and independent of chart
complexity, which is what removes the original timeout risk.

### Annotation on the instance

Annotation key: `kubeplus.io/node-isolation`, placed on the Kind instance CR (unchanged
from the earlier annotation-based design, minus the `mode` field, since this issue covers
dedicated placement only):

```yaml
apiVersion: cloudark.io/v1
kind: Agent
metadata:
name: team-a-agent-instance
namespace: team-a
annotations:
kubeplus.io/node-isolation: |
{
"nodeSelector": {"pool": "team-a-pool"},
"tolerations": [
{"key": "dedicated", "operator": "Equal", "value": "team-a-pool", "effect": "NoSchedule"}
]
}
```

When KubePlus creates (or labels) the instance's namespace, it copies this resolved value
onto the Namespace object itself:

```yaml
apiVersion: v1
kind: Namespace
metadata:
name: team-a
labels:
kubeplus.io/node-isolation-enabled: "true"
annotations:
kubeplus.io/node-isolation-resolved: |
{
"nodeSelector": {"pool": "team-a-pool"},
"tolerations": [
{"key": "dedicated", "operator": "Equal", "value": "team-a-pool", "effect": "NoSchedule"}
]
}
```

The label exists purely so `namespaceSelector` can match on it cheaply; the annotation
carries the actual values the webhook needs, so the webhook never has to read the
instance CR at request time — only the Namespace, via its own cache.

### Mutating webhook configuration changes

This is a new, independent entry in the `webhooks` list — it does not modify or interact
with the existing CRD-focused rule (`workflows.kubeplus`/`platformapi.kubeplus`), since
that rule's `apiGroups` don't include core resources and therefore can't match Pods
regardless of `resources: ["*"]`.

```json
{
"name": "node-isolation.kubeplus.io",
"rules": [
{
"apiGroups": [""],
"apiVersions": ["v1"],
"operations": ["CREATE"],
"resources": ["pods"],
"scope": "Namespaced"
}
],
"namespaceSelector": {
"matchExpressions": [
{
"key": "kubeplus.io/node-isolation-enabled",
"operator": "In",
"values": ["true"]
}
]
},
"failurePolicy": "Ignore",
"sideEffects": "None",
"admissionReviewVersions": ["v1"]
}
```

This `namespaceSelector` expression is static — written once, at KubePlus install time, as
part of the same Helm chart that already configures the CRD rule. It never needs to be
edited per instance. What changes per instance is only the `kubeplus.io/node-isolation-enabled`
label on the relevant Namespace object, which is set and removed as a side effect of that
namespace's own create/delete lifecycle — there is no separate reconciliation of the
webhook config itself.

`failurePolicy: Ignore` is deliberate: if the webhook is briefly unavailable, pods should
still be admitted (fail open) rather than blocking all pod creation in the cluster because
of an isolation feature being temporarily down. Isolation strength here is a placement
guarantee, not a security boundary of last resort — losing it briefly during webhook
downtime is an acceptable trade for not taking down pod scheduling cluster-wide.

### JSON patch implementation (Go)

```go
package nodeisolation

import (
"encoding/json"

corev1 "k8s.io/api/core/v1"
)

// NodeIsolationSpec is the resolved value copied onto a Namespace's
// kubeplus.io/node-isolation-resolved annotation.
type NodeIsolationSpec struct {
NodeSelector map[string]string `json:"nodeSelector"`
Tolerations []corev1.Toleration `json:"tolerations"`
}

// namespaceCache is populated by a Namespace informer at webhook startup and
// kept current via Add/Update/Delete event handlers. Lookups are in-memory
// map reads keyed by namespace name — no API calls happen per admission
// request.
type namespaceCache interface {
Get(namespace string) (*NodeIsolationSpec, bool)
}

// BuildNodeIsolationPatch returns a JSON Patch (RFC 6902) that merges the
// namespace's required nodeSelector/tolerations into the incoming Pod's spec.
// Required values always win over anything a chart's own values.yaml might
// already have set, but existing entries the isolation policy doesn't
// mention are preserved rather than clobbered.
func BuildNodeIsolationPatch(pod *corev1.Pod, iso *NodeIsolationSpec) ([]byte, error) {
var patches []map[string]interface{}

if len(iso.NodeSelector) > 0 {
merged := make(map[string]string, len(pod.Spec.NodeSelector)+len(iso.NodeSelector))
for k, v := range pod.Spec.NodeSelector {
merged[k] = v
}
for k, v := range iso.NodeSelector {
merged[k] = v // isolation-mandated keys take precedence
}

op := "add"
if len(pod.Spec.NodeSelector) > 0 {
op = "replace"
}
patches = append(patches, map[string]interface{}{
"op": op,
"path": "/spec/nodeSelector",
"value": merged,
})
}

if len(iso.Tolerations) > 0 {
if len(pod.Spec.Tolerations) == 0 {
// No existing tolerations array — add the whole thing.
patches = append(patches, map[string]interface{}{
"op": "add",
"path": "/spec/tolerations",
"value": iso.Tolerations,
})
} else {
// Append to the existing array rather than clobbering
// whatever tolerations the chart itself declared.
for _, t := range iso.Tolerations {
patches = append(patches, map[string]interface{}{
"op": "add",
"path": "/spec/tolerations/-",
"value": t,
})
}
}
}

if len(patches) == 0 {
return nil, nil
}
return json.Marshal(patches)
}

// HandleAdmission is the webhook's per-request entry point. Given the
// existing namespaceSelector-based filtering at the MutatingWebhookConfiguration
// level, every request reaching here is already known to be a Pod CREATE in a
// namespace labeled kubeplus.io/node-isolation-enabled=true.
func HandleAdmission(namespace string, pod *corev1.Pod, cache namespaceCache) (patch []byte, err error) {
iso, ok := cache.Get(namespace)
if !ok {
// Label present but annotation missing/unparseable — fail open,
// consistent with failurePolicy: Ignore at the config level.
return nil, nil
}
return BuildNodeIsolationPatch(pod, iso)
}
```

The response wrapping (`admissionv1.AdmissionResponse{Patch: patch, PatchType: &jsonPatchType}`)
is omitted here since it's boilerplate already present in KubePlus's existing webhook
server — `HandleAdmission` is the only new logic that needs to be wired into it, plus the
Namespace informer/cache used by `namespaceCache.Get`.

### Acceptance criteria

- [ ] The new Pod-targeting webhook rule is a separate `webhooks[]` entry from the
existing CRD rule; neither rule's behavior changes as a result of adding the other.
- [ ] A Pod created in a namespace without `kubeplus.io/node-isolation-enabled: "true"`
is never sent to this webhook logic at all (verify via webhook server request
logs/metrics showing zero invocations for unrelated namespaces).
- [ ] A Pod created in an isolated namespace receives the mandated `nodeSelector` merged
with (not replacing) any selector the chart's own template already set, with
isolation-mandated keys winning on conflict.
- [ ] A Pod created in an isolated namespace receives the mandated `tolerations`
appended to (not replacing) any tolerations the chart's own template already set.
- [ ] Webhook latency for Pod admission in an isolated namespace does not meaningfully
increase with chart complexity (test with a chart producing pods through a
multi-level CR chain vs. a plain Deployment — latency should be comparable, since
no traversal happens either way).
- [ ] Deleting the instance (and its namespace) removes the label naturally; no explicit
unlabeling step is required or should be added, since a namespace and its instance
share a lifecycle in the current one-namespace-per-instance model.

### Demo steps

1. Taint two nodes and label them for a dedicated pool:
```bash
kubectl taint nodes node-a dedicated=team-a-pool:NoSchedule
kubectl label nodes node-a pool=team-a-pool
```
2. Instantiate an Agent Kind instance with:
```yaml
metadata:
annotations:
kubeplus.io/node-isolation: |
{"nodeSelector": {"pool": "team-a-pool"},
"tolerations": [{"key": "dedicated", "operator": "Equal", "value": "team-a-pool", "effect": "NoSchedule"}]}
```
3. Confirm the namespace carries both the label and the resolved annotation:
```bash
kubectl get namespace team-a -o jsonpath='{.metadata.labels}{"\n"}{.metadata.annotations}'
```
4. Confirm the resulting pods carry the merged `nodeSelector`/`tolerations` and land only
on `node-a`, without the chart itself having declared either field:
```bash
kubectl get pods -n team-a -o jsonpath='{.items[*].spec.nodeSelector}'
kubectl get pods -n team-a -o wide
```
5. Create an unrelated Pod in a namespace without the label and confirm (via webhook
server logs) that the admission request never reached the node-isolation handler at
all.
6. Delete the instance and confirm the namespace (and therefore the label) is gone, with
no separate cleanup step required:
```bash
kubectl delete agent team-a-agent-instance -n team-a
kubectl get namespace team-a # expect: not found
```

No repository file paths or tests are named. Start at the existing webhook server and Helm chart entry that configures the CRD rule, then trace namespace annotation handling and admission registration. Wire the namespace cache and Pod handler, and verify the separate rule, merged selector and tolerations, fail-open behavior, latency independence, and namespace lifecycle using the stated acceptance and demo checks.

Written by the indexing model from the issue text.