[Bug]: gateway-controller Deployment defaults to RollingUpdate while mounting a ReadWriteOnce PVC, deadlocking upgrades on multi-node clusters

Please select the area the issue is related to

Gateway

Please select the aspect the issue is related to

Aspect/Configuration (Config files, settings, env vars, defaults)

Description

gateway-helm-chart renders the gateway-controller Deployment with no strategy, so Kubernetes applies the default RollingUpdate. The same chart gives the controller a single replica and a ReadWriteOnce PVC for its SQLite store. On a multi-node cluster those three defaults deadlock every upgrade.

At replicas: 1 the defaults resolve to maxSurge=1, maxUnavailable=0, so the replacement pod must reach Ready before the old pod may be evicted. The scheduler's spread scoring prefers a node that does not already host a replica of the same owner, so the new pod usually lands on a different node and cannot attach the volume:

Warning  FailedAttachVolume  Multi-Attach error for volume "pvc-…"
  Volume is already used by pod(s) …-gw-gateway-controller-…

The new pod waits for a detach, the detach waits for the old pod to go away, and the old pod cannot go away until the new pod is Ready. Nothing breaks the cycle: progressDeadlineSeconds only marks the rollout failed, it does not abort or roll back. The APIGateway stays at phase: Reconciling / Programmed=False indefinitely.

Single-node clusters are unaffected, because ReadWriteOnce permits multiple pods on the same node. That is why this does not appear on local/dev installs and only surfaces the first time a gateway is upgraded on a real cluster.

Relevant chart source (1.2.0-beta; identical in 1.1.5, 1.2.0-alpha and 1.2.0-rc):

templates/gateway/controller/deployment.yaml

kind: Deployment
  replicas: {{ $deployment.replicaCount }}
  {{- with $deployment.strategy }}
  strategy:
    {{- toYaml . | nindent 4 }}
  {{- end }}

values.yaml

    accessModes:
      - ReadWriteOnce
    size: 100Mi
    replicaCount: 1
    # Deployment update strategy (full apps/v1 DeploymentStrategy object).
    # Empty = Kubernetes default (RollingUpdate, 25% maxSurge / 25% maxUnavailable).
    strategy: {}

Because strategy is empty, the with guard omits the key entirely and the API server defaults it.

There is a second-order effect: gateway-operator installs this sub-release with wait enabled and a 300s timeout, so the stall also fails the operator's Helm operation and feeds its retry path. That can leave the release in pending-install and the APIGateway unrecoverable without deleting the CR. Filed separately.

Steps to Reproduce

1. Create a Kubernetes cluster with more than one node and a ReadWriteOnce default StorageClass (any managed provider's default block storage qualifies).
2. Install gateway-operator 0.10.1 and have it deploy gateway 1.2.0-beta (or 1.2.0-rc) with the controller's default persistence, i.e. storage.type: sqlite and persistence.enabled: true.
3. Wait for the gateway to reach Programmed=True and note which node the controller pod is on.
4. Trigger any change that causes the operator to upgrade the gateway sub-release — for example altering a value in the APIGateway's configRef ConfigMap.
5. Observe two controller pods: the original Running, the replacement stuck in ContainerCreating.

kubectl get pods -n <gateway-namespace> -o wide | grep gateway-controller
kubectl describe pod <new-controller-pod> -n <gateway-namespace> | sed -n '/Events:/,$p'
kubectl get apigateway <name> -n <gateway-namespace> \
  -o jsonpath='phase={.status.phase} Programmed={.status.conditions[?(@.type=="Programmed")].status}{"\n"}'

The describe output shows Multi-Attach error for volume, and the APIGateway reports phase=Reconciling, Programmed=False. It does not recover; deleting the old controller pod releases the volume and the rollout then completes.

Confirming the two contributing defaults:

kubectl get deploy <release>-gateway-controller -n <gateway-namespace> \
  -o jsonpath='{.spec.strategy.type}{"\n"}'          # RollingUpdate
kubectl get pvc <release>-gateway-controller-data -n <gateway-namespace> \
  -o jsonpath='{.spec.accessModes[*]}{"\n"}'          # ReadWriteOnce

Severity Level of the Issue

Severity/Critical (Core functionality is broken but there is a workaround. Need urgent attention)

Environment Details (with versions)

• Kubernetes 1.35, 3 nodes, ReadWriteOnce default StorageClass (CSI block storage)
• gateway-operator chart 0.10.1
• gateway chart 1.2.0-beta, controller with storage.type: sqlite, persistence.enabled: true, replicaCount: 1

Checked against every published tag of gateway — 1.1.5, 1.2.0-alpha, 1.2.0-beta, 1.2.0-rc — and against main: deployment.yaml and the strategy: {} default are identical in all of them, so the newest available gateway is still affected.

Suggested fix

Default the controller to strategy: {type: Recreate} when it owns a ReadWriteOnce volume — i.e. whenever persistence.enabled is true and storage.type is sqlite. The controller cannot exceed one replica in that configuration, so Recreate gives up no availability; it only trades a few seconds of downtime per upgrade for a rollout that completes unattended. A StatefulSet would also solve it and is arguably the better shape, but it is a larger change and no StatefulSet option exists today.

Two notes for whoever picks this up:

• gateway.controller.deployment.strategy already accepts a full DeploymentStrategy and renders correctly (added in #2179), so this is a default-value change rather than new plumbing. Consumers can set it themselves in the meantime.
• persistence.enabled=false is not a viable workaround: it switches the volume to emptyDir, and #2147 documents the result — once the SQLite store is wiped, every RestApi route returns 404 while the CRs still report Programmed=True, with no self-healing.

Start with templates/gateway/controller/deployment.yaml and the controller section of values.yaml, then reproduce the rollout on a multi-node cluster using the kubectl commands in the issue. Verify that the rendered Deployment uses a safe strategy for the SQLite and ReadWriteOnce configuration, while preserving the existing strategy override and confirming upgrades complete without a Multi-Attach stall.