[Bug]: APIGateway is permanently wedged after retry exhaustion, and a sub-release stuck in pending-install is never recovered

Please select the area the issue is related to

Gateway Operator

Please select the aspect the issue is related to

Aspect/Other (Anything else which does not match above categories)

Description

When a gateway's first install fails, the operator can leave the APIGateway permanently wedged. Two independent defects combine, and each is worth fixing on its own.

1. A sub-release stuck in pending-install is never recovered. internal/helm/client.go decides between install and upgrade purely on whether history exists:

histClient := action.NewHistory(actionConfig)
histClient.Max = 1
_, err = histClient.Run(opts.ReleaseName)
releaseExists := err == nil

if releaseExists {
    log.Info("Release exists, performing upgrade", ...)
    return c.upgrade(ctx, actionConfig, opts)
}
return c.install(ctx, actionConfig, opts)

A release stuck in pending-install at revision 1 does have history, so releaseExists is true and the operator calls upgrade, which Helm refuses with another operation (install/upgrade/rollback) is in progress. There is no handling of pending-* states anywhere — no rollback, no Atomic, no CleanupOnFail, no status check before deciding. IsReleaseDeployed does inspect rel.Info.Status, but it is only used for reporting, never in this decision.

Note the misleading symptom: another operation in progress reads like a concurrency problem, when it is actually the operator repeatedly choosing the wrong Helm operation for a release that never completed.

How the release reaches that state: deploy.go passes Wait: true with a 300s timeout and calls client.Run(...) rather than RunWithContext, so the install blocks non-cancellably. A pending-install at revision 1 means that process was killed mid-install (pod restart, OOM, eviction, lost leader election) rather than failing cleanly — a clean wait-timeout would leave the release failed, not pending.

2. After retry exhaustion the CR cannot be re-driven. The retry counter itself is in-memory (GatewayTracker) and is correctly cleared by an operator restart. The problem is what gets persisted. handleGatewayDeploymentError writes the exhaustion into the CR:

if entry.RetryCount >= maxRetries {
    entry.Status = GatewayTrackingStatusDeployed
    ...
    Status:             metav1.ConditionFalse,
    ObservedGeneration: entry.Generation,
    Reason:             GatewayProgrammedReasonDeploymentFailed,
    Message:            fmt.Sprintf("Max retries (%d) exceeded. Last error: %s", ...),

Reconcile then reads statusObservedGen from that condition, and decideAndProcess has two live branches: one requiring Programmed == True, the other gated on crGeneration > statusObservedGen. After exhaustion at generation 1 the CR has Programmed=False and observedGeneration == crGeneration, so both gates fail and control falls through to the default "nothing to do" return.

The consequence is that none of the obvious remedies work, and none of them report why:

Recovery required deleting the sub-release with helm uninstall and deleting the APIGateway CR so it could be re-rendered.

Steps to Reproduce

1. Install gateway-operator 0.10.1.
2. Create an APIGateway whose gateway sub-release cannot start — for example with gateway.controller.encryptionKeys.enabled=true pointing at a Secret that does not exist, so the controller crash-loops and the operator's Wait: true install cannot converge.
3. While that first install is in flight (within its 300s window), kill the operator pod so the install is interrupted rather than failing cleanly:

   kubectl delete pod -l app.kubernetes.io/name=gateway-operator -n <ns>
   

4. Confirm the sub-release is stranded:

   helm list -a -n <ns>          # <release>-gw shows pending-install at revision 1
   

5. Let the operator retry until exhaustion (~4 minutes of backoff: 1+2+4+8+16+32+60+60+60s), then observe:

   kubectl get apigateway <name> -n <ns> -o jsonpath='{.status.conditions[?(@.type=="Programmed")]}' | jq
   

   status: "False", reason: DeploymentFailed, message Max retries (10) exceeded. Last error: … another operation (install/upgrade/rollback) is in progress, and observedGeneration equal to the CR's metadata.generation.
6. Now fix the underlying cause (create the missing Secret), and try each of: restarting the operator, annotating the CR, editing the configRef ConfigMap, and waiting out the resync period. None changes the status, and nothing is logged above debug level to explain why.

Related: on a multi-node cluster this is easy to reach without step 3, because the controller's rolling update deadlocks on its ReadWriteOnce volume (filed separately). That stall makes the operator's Wait: true Helm operation fail and feeds the same retry path, so the two are very likely the same incident observed at two stages.

Severity Level of the Issue

Severity/Major (Important functionality is broken. Should be prioritized. Doesn't need immediate attention)

Environment Details (with versions)

• Kubernetes 1.35, 3 nodes
• gateway-operator chart 0.10.1 (newest published; equals main)
• gateway chart 1.2.0-beta
• Operator config: max_retry_attempts: 10, initial_backoff: 1s, max_backoff_duration: 60s, sync_period: 10m

Suggested fix

For (1) — in InstallOrUpgrade, inspect rel.Info.Status rather than only asking whether history exists. For pending-install at revision 1 the release never deployed, so uninstall-then-install is safe; for pending-upgrade/pending-rollback a Rollback to the last deployed revision is appropriate. Setting CleanupOnFail and using RunWithContext so the operation is cancellable would also prevent new occurrences.

For (2) — do not let the crGeneration > statusObservedGen gate swallow a failed CR. Either exclude Programmed=False/DeploymentFailed from that gate so a resync, an operator restart or a corrected configRef can re-drive it, or stop writing ObservedGeneration on the exhaustion path so the CR remains eligible for reconcile. A user-visible reset path would help regardless — today the only documented remedy is deleting the CR, and the two undocumented ones (a real spec change to bump metadata.generation, or patching the status subresource and restarting the operator) are only discoverable by reading the controller source.

Start with internal/helm/client.go and deploy.go, then trace handleGatewayDeploymentError and decideAndProcess in the gateway reconciliation path. Reproduce the interrupted pending-install and retry-exhaustion scenarios described in the issue; done means pending Helm releases can recover and a failed APIGateway can be re-driven after restart or correction without deleting the CR.