Platform hub: system-peeks NodePool fragility — ineffective disruption budget + single-node hard-pinning cause cascading outage on Auto Mode disruption

## Summary

On the platform hub (EKS Auto Mode), a **forced/involuntary node disruption** of the single `system-peeks` node caused a **cascading outage**: the node got stuck `Draining` for ~25h, all critical system add-ons hard-pinned to `system-peeks` went `Pending`, External Secrets went down (its `SecretStore` never became `Valid`), and **Keycloak never deployed** (blocked waiting on `SecretStore/argocd-store`) → `https:///keycloak/...` returned 404.

Recovery required manually deleting the `system-peeks` NodePool (ArgoCD `selfHeal` recreated it clean) + removing the stuck NodeClaim finalizer, after which Karpenter provisioned 2 fresh nodes across 2 AZs and the whole stack recovered.

This issue lists the underlying problems to fix. A follow-up PR will implement the fixes.

## Incident evidence

- NodeClaim `system-peeks-` had `deletionTimestamp` set (marked for disruption) but stayed `Ready` + `Drained=Unknown (Draining)` for ~25h — never terminated.
- CloudTrail around the same time shows a wave of `TerminateInstances` by `eks-auto-mode-compute-*` (Auto Mode controller), i.e. an **automated** disruption event, not a manual teardown (CFN stack was `CREATE_COMPLETE` throughout).
- 25–28 pods stuck `Pending` with `FailedScheduling` on `karpenter.sh/nodepool In [system-peeks]`; Auto Mode did **not** provision a replacement `system-peeks` node.

## Problem 1 — Disruption budget does not do what it was designed to do

Current `system-peeks` (and the other `-peeks` NodePools) disruption config:

```yaml
disruption:
consolidationPolicy: WhenEmptyOrUnderutilized
consolidateAfter: 5m
budgets:
- nodes: "0" # no schedule -> ALWAYS active
- nodes: "30%"
schedule: "0 20 * * *"
duration: 2h
```

**Intent:** allow Karpenter voluntary consolidation only during a daily maintenance window (20:00–22:00 UTC), forbid it the rest of the time — to regulate churn when we over-deploy.

**Actual behaviour:** a budget with no `schedule` is active 24/7, and Karpenter applies the **most restrictive** active budget. So the always-on `nodes: "0"` clamps the effective allowance to **0 at all times** — the scheduled `30%` window never wins. Net result: **voluntary consolidation is blocked 24/7** and the maintenance window is a no-op.

**Fix direction:** express the block as a *scheduled* budget covering the non-window hours (so only one budget is active at a time), e.g. keep `nodes: "0"` on a schedule/duration that covers 22:00→20:00, and `nodes: "30%"` for 20:00–22:00. To be validated against the pinned Karpenter/Auto Mode version's budget semantics.

## Problem 2 — Single-node `system-peeks` + hard pinning ⇒ drain deadlock on any disruption

- `system-peeks` typically runs a **single node** at steady state (one node satisfies the pinned workloads).
- Critical add-ons are pinned with a **hard** `nodeSelector: { karpenter.sh/nodepool: system-peeks }` (they require amd64; the built-in `system` pool provisions arm64/Graviton).
- When that single node is disrupted (see Problem 5), its pods have **no other `system-peeks` node** to move to → the NodeClaim gets stuck in `Draining` (never completes termination), and the pinned pods sit `Pending`.

Important: **disruption budgets gate node *removal*, never node *creation*.** So the budget did not prevent a replacement — the replacement failed because of the drain/scheduling deadlock (single node + hard pin, amplified by the stuck NodeClaim). This is the core fragility.

**Fix direction:** guarantee `system-peeks` always has **≥2 nodes spread across ≥2 AZs** so any disruption has a landing zone (see Problem 3 + Problem 6).

## Problem 3 — HA gaps in add-ons pinned to `system-peeks`

Not all critical add-ons are HA, and some have 2 replicas but **no `topologySpreadConstraints`** (both replicas can land on the same node). Observed on the hub:

| Workload | ns | replicas | zone topologySpread |
|---|---|---:|:--:|
| cert-manager (+cainjector, +webhook) | cert-manager | 2 | ✅ |
| crossplane (+rbac-manager) | crossplane-system | 2 | ✅ |
| external-secrets (+cert-controller, +webhook) | external-secrets | 2 | ✅ |
| keycloak (StatefulSet) | keycloak | 2 | ✅ |
| **aws-load-balancer-controller** | kube-system | 2 | ❌ no spread |
| **metrics-server** | kube-system | 2 | ❌ no spread |
| **external-dns** | kube-system | 1 | ❌ |
| **kube-state-metrics** | kube-prometheus-stack | 1 | ❌ |
| **argo-events-controller-manager** | argo-events | 1 | ❌ |
| **argo-server / workflow-controller** | argo | 1 | ❌ |
| **backstage** | backstage | 1 | (spread set, 1 replica) |
| **postgresql (backstage)** | backstage | 1 | ❌ (stateful) |
| **postgresql (keycloak)** | keycloak | 1 | ❌ (stateful) |

Note: even the HA-with-spread add-ons only stay resilient if `system-peeks` maintains **≥2 nodes in ≥2 AZs** — with `whenUnsatisfiable: DoNotSchedule` and a single node, the second replica cannot schedule.

**Fix direction:** add zone `topologySpreadConstraints` to LBC and metrics-server; raise single-replica critical controllers to 2 where supported; decide an HA story for the Postgres StatefulSets (backstage, keycloak).

## Problem 4 — Guarantee a minimum number of `system-peeks` nodes

There is no first-class "minimum nodes per NodePool" in the Karpenter `NodePool` CRD (`spec.limits` is a max only). Options to evaluate:

- A low-priority **balloon/placeholder Deployment** (2 replicas, `podAntiAffinity` by hostname, pinned to `system-peeks`) to keep a warm 2nd node across AZs.
- Confirm whether the pinned Karpenter/EKS Auto Mode version exposes any static/minimum-capacity capability we can use instead.
- Combined with proper `topologySpreadConstraints` on the critical add-ons (Problem 3), which already imply ≥2 nodes across AZs when enforced.

## Problem 5 — Node auto-repair does not cover the `-peeks` NodePools

Cluster `computeConfig` scopes managed compute / repair to the built-in pools only:

```json
{ "enabled": true, "nodePools": ["system", "general-purpose"], "nodeRoleArn": "..." }
```

The custom `-peeks` NodePools (`system-peeks`, `general-purpose-peeks`, `gpu`, `trainium`) are not in this list.

**Fix direction:** investigate how to extend EKS Auto Mode node auto-repair to the custom `-peeks` NodePools, and where that is configured. **Only enable this after Problem 2/3/4 are fixed** — auto-repairing a single-node pinned pool with no landing zone would recreate the same deadlock.

## Key takeaway

Disruption budgets reduce *voluntary* churn but are **not a resilience mechanism** — **forced/involuntary disruptions bypass budgets by design** (node health/auto-repair, instance degradation, forced image updates). Resilience against those comes from **redundancy** (≥2 nodes / ≥2 AZs + properly spread HA workloads), not from blocking disruption.

## Affected files (appmod-blueprints)

- `gitops/addons/charts/platform-manifests-bootstrap/templates/custom-nodepools.yaml` (the `-peeks` NodePool definitions + disruption budgets)
- Add-on charts/values under `gitops/addons/**` for the workloads pinned to `system-peeks` (replicas / `topologySpreadConstraints`)

## Environment

- EKS Auto Mode, Kubernetes v1.35, Karpenter `karpenter.sh/v1` NodePools, Bottlerocket (EKS Auto) nodes.
- Add-on NodeClass: `default` (Auto Mode).

Start with gitops/addons/charts/platform-manifests-bootstrap/templates/custom-nodepools.yaml to inspect the -peeks NodePools and disruption budgets. Then trace the add-on charts and values under gitops/addons/** for workloads pinned to system-peeks, and verify the pinned Karpenter/EKS Auto Mode version's budget and repair semantics. Done means the affected pools and add-ons have a validated resilience design for disruption, node redundancy, spreading, and repair.

Written by the indexing model from the issue text.