`helm upgrade` causes a full data-plane outage: DaemonSet instances report Ready before the scheduler configures them

We've finding that we can't seamlessly deploy Bunkerweb upgrades due to the issue outlined below. I hope you don't mind Claude having put together the details of what we've discovered!

Thanks very much...

## Summary

On every `helm upgrade` of a DaemonSet deployment, we get a **~5–7 minute full outage across all sites** while the chart rolls. The browser symptom is `PR_END_OF_FILE_ERROR` (TLS connection closed by the default server). It self-heals once the scheduler re-pushes config to the new instance pods — no manual action needed — but there is a real window where **every** BunkerWeb instance is serving only the default server.

Root cause is a combination of (1) instances holding config only in ephemeral container storage with no bootstrap-on-start, (2) the instance readiness probe reporting `Ready` before any real config is loaded, and (3) the control plane (scheduler/controller) rolling at the same time as the data plane. The readiness gap is the key one: because a config-less instance is marked `Ready`, the DaemonSet `RollingUpdate` marches through every node faster than the scheduler can re-configure them.

## Environment

| | |
|---|---|
| Chart | `bunkerweb-helm` 1.0.26 (app 1.6.14); also seen on 1.0.16 / 1.6.9 |
| Mode | `bunkerweb.kind: DaemonSet`, `hostPorts: true`, Kubernetes autoconf controller + scheduler |
| Service | `type: LoadBalancer`, `externalTrafficPolicy: Local`, PROXY protocol v2 (OVH cloud LB) |
| State | external managed MySQL, built-in Redis |
| Relevant settings | `DISABLE_DEFAULT_SERVER=yes`, `DISABLE_DEFAULT_SERVER_STRICT_SNI=yes`, `USE_PROXY_PROTOCOL=yes` |

## What happens

During `helm upgrade`, the instance DaemonSet, the scheduler, and the controller all roll together. Observed in the scheduler log — it keeps trying to push to the **old** (terminated) instance pod IPs, on serial 5s connect timeouts:

```
[API.CALLER] [❌] - Can't send API request to http://10-2-1-20.bunkerweb.pod.cluster.local:5000/custom_configs : ... connect timeout=5
[SCHEDULER] [❌] - Error while sending /etc/bunkerweb/configs to BunkerWeb instances
[SCHEDULER] [⚠️] - No BunkerWeb instance found, skipping bunkerweb reload ...
```

Meanwhile the **new** instance pods are `Ready` and receiving traffic while serving only the default server, so with strict-SNI they close every TLS handshake:

```
_ 10.x.x.x - ... "\x00" 400 ... # default server rejecting; this is what users hit
```

Several minutes later the scheduler finally discovers the new pod IPs and recovers:

```
[API.CALLER] [ℹ️] - Successfully sent API request to http://10-2-1-26.bunkerweb.pod.cluster.local:5000/custom_configs
[SCHEDULER] [ℹ️] - Successfully sent /etc/bunkerweb/configs folder to reachable BunkerWeb instances
```

## Root cause analysis

1. **Instances have no config until pushed, and don't self-bootstrap.** Instance config lives in the container's ephemeral `/etc/nginx` and is delivered by the scheduler over the `:5000` API. A freshly-started pod serves nothing until that push arrives; it does not restore last-known config from the shared database on startup.

2. **Readiness does not reflect config state.** The readiness probe runs `healthcheck.sh ok`, which passes as soon as nginx is up and `/healthz` returns `ok` — **this is true for a config-less instance serving only the default server.** So the pod is added to the Service endpoints / LB rotation while it cannot actually serve any site. This also lets the DaemonSet `RollingUpdate` (`maxUnavailable: 1`) advance node-by-node in ~30s each, so all N instances become config-less within a couple of minutes.

3. **Control plane and data plane roll simultaneously.** The scheduler restarts in the same release and comes back with a stale instance list, burning minutes on serial 5s timeouts to dead pod IPs. So the data plane goes fully blank *faster* than the control plane can re-push. The outage is the gap between the two.

## Impact

- Full loss of service for all sites for several minutes on **every** upgrade (and on any event that rolls the instance pods).
- On shared clusters this hits **all tenants** at once (we host multiple independent sites behind one BunkerWeb).
- The failure mode is silent to Kubernetes: pods are `Ready`, the rollout "succeeds", but the edge is down — so it won't trip `--wait`/`--atomic` protections in the obvious way.

## Proposed fixes (in priority order)

**1. Readiness should reflect config-loaded state (core app — `bunkerity/bunkerweb`). Highest impact.**
An instance serving only the default server should report **NotReady** until it has received/loaded a real (non-default) configuration. This single change makes Kubernetes-native rollouts safe by construction: the Service/LB won't route to unconfigured instances, and the DaemonSet won't advance past a node until its instance is genuinely serving. Today `/healthz` returns `ok` for a config-less instance, which is the crux of the outage.

**2. Instances should bootstrap last-known config from the shared DB on startup (core app).**
So a replaced pod serves immediately, and a scheduler hiccup during a roll can't black-hole traffic. This also removes the push-timing dependency entirely.

**3. Chart: sequence the rollout and expose the knobs (`bunkerweb-helm`).**
- Roll scheduler/controller/redis **before** the instance DaemonSet (hook weights or an init-gate), rather than all at once.
- Expose `bunkerweb.updateStrategy` (including `OnDelete`), `bunkerweb.minReadySeconds`, and a PodDisruptionBudget for the instance DaemonSet. None are configurable today, so operators can't tune a safe rollout without `kubectl patch`-ing live objects (which then drifts from the release).

**4. Scheduler: reconverge the instance set quickly (core app).**
On a roll it should drop unreachable instances fast / refresh the list promptly, and use shorter or parallel connect timeouts instead of serial 5s waits on dead pods.

Fix **1 by itself** would have prevented this outage.