Controller revoked all shared backend SG rules during pod churn (stale SG cache race), causing ~45 min cluster-wide outage; checkpoint-hash skip delayed self-repair by 30 min

**Bug Description**

**EDIT:** root-cause mechanism refined — see comment below

During routine Fargate pod recycling, two TargetGroupBinding reconciles raced on the 10-minute-TTL SecurityGroup info cache and the controller **removed every ingress rule** granting the shared backend SecurityGroup (`elbv2.k8s.aws/targetGroupBinding=shared`) access to the EKS cluster security group. CloudTrail shows the controller **revoking a rule it had authorized 45 seconds earlier, with no replacement grant**. Every ALB in the VPC (4 ingress groups, all environments) lost connectivity to all targets simultaneously — failing health checks and HTTP 504 for ~45 minutes.

Self-repair was then delayed a further 30 minutes: the controller pods were recycled by the same Fargate maintenance wave, and the new leader **skipped reconciliation of every TGB because the checkpoint hash matched** (`"Skipping targetgroupbinding reconcile"`). The hash reflects the K8s-side spec, not actual AWS SG state, so the missing rules went undetected until the deferred TGB reconciler ran ~30 minutes after startup.

This appears to be the same failure class previously reported in #3637 (identical CloudTrail signature, closed unreproduced), #3639 (revoke/authorize window on the shared rule, acked `/kind bug`, closed by staleness bot), #2578, and #3771. This report includes the CloudTrail evidence and controller logs that were requested in #3637, plus a code-path analysis at the exact commit we run.

**Steps to Reproduce**

- Step-by-step guide to reproduce the bug:

This is a race condition, so it is not deterministic, but the conditions are simple:

1. Cluster with `ip` target type (ours is all-Fargate), shared backend SG and restricted SG rules (both defaults), and ≥2 TargetGroupBindings whose target ports define the aggregated port range — e.g. most apps on port 3000/3008 and one app (mailpit) on 8025, producing a single `tcp 3000-8025` rule on the cluster SG.
2. Replace the pod backing the TGB that defines the **max** port of the range (8025), so its endpoint deregisters and re-registers within the 10-minute SG info cache TTL (`defaultSGInfoCacheTTL`, [security_group_manager.go#L19](https://github.com/kubernetes-sigs/aws-load-balancer-controller/blob/3f1e9707ac4ce65504d72b425e82be5245b38e0e/pkg/networking/security_group_manager.go#L19)). In our case AWS Fargate maintenance recycled the pod; a plain `kubectl delete pod` should exercise the same path.
3. Race window: the endpoint-remove reconcile shrinks the aggregate (authorize `3000-3008`, revoke `3000-8025`); the endpoint-add reconcile then runs with a pre-shrink **cached** view of the SG. Diffing desired (`3000-8025`) against the stale cache (which still contains `3000-8025` *and* `3000-3008`) yields: grant **nothing**, revoke `3000-3008`. The revoke succeeds, so the error-triggered cache bypass ([security_group_reconciler.go#L81-L92](https://github.com/kubernetes-sigs/aws-load-balancer-controller/blob/3f1e9707ac4ce65504d72b425e82be5245b38e0e/pkg/networking/security_group_reconciler.go#L81-L92)) never fires. Net AWS state: zero rules.

- Manifests applied while reproducing the issue:

No manifests were applied — the incident was triggered by AWS-side Fargate pod recycling with no cluster changes in the preceding ~19 hours. The ingresses are unremarkable host-based ALB ingresses (see Environment below); TargetGroupBindings are all controller-created from those ingresses.

- Controller logs/error messages while reproducing the issue:

The acting pod's logs are unavailable — it was removed by the same Fargate recycling wave minutes after the incident (this likely also explains why #3637's reporter could not supply logs: the trigger destroys the acting pod). **CloudTrail is the authoritative record of its actions** (timestamps UTC, account redacted):

| Time | CloudTrail event | Result |
|---|---|---|
| 09:55:16 | `AuthorizeSecurityGroupIngress` `tcp 3000-3008` on cluster SG | OK |
| 09:55:17 | `DeregisterTargets` (replaced pod's old IP) on the port-8025 TG | OK |
| 09:55:17 | `RevokeSecurityGroupIngress` `tcp 3000-8025` | OK |
| 09:55:17, 09:55:18 | Two more revokes of `tcp 3000-8025` | `Client.InvalidPermission.NotFound` — concurrent stale-cache reconciles |
| **09:56:01** | **`RevokeSecurityGroupIngress` `tcp 3000-3008` — the rule authorized at 09:55:16, no replacement granted. Zero rules remain.** | **OK** |
| 09:56:02 | `RegisterTargets` (replacement pod's new IP) on the port-8025 TG | OK |

The critical event in full (redacted):

```json
{
"eventTime": "2026-08-28T09:56:01Z",
"eventSource": "ec2.amazonaws.com",
"eventName": "RevokeSecurityGroupIngress",
"userAgent": "elbv2.k8s.aws/v3.4.0",
"requestParameters": {
"groupId": "sg-053bdd55831dccd60",
"ipPermissions": { "items": [{
"ipProtocol": "tcp", "fromPort": 3000, "toPort": 3008,
"groups": { "items": [{
"groupId": "sg-014f7b94f96fe2d8d",
"description": "elbv2.k8s.aws/targetGroupBinding=shared"
}]}
}]}
},
"responseElements": {
"_return": true,
"revokedSecurityGroupRuleSet": { "items": [{
"groupId": "sg-053bdd55831dccd60",
"securityGroupRuleId": "sgr-0429e9c3bf232a86c",
"isEgress": false, "ipProtocol": "tcp",
"fromPort": 3000, "toPort": 3008,
"referencedGroupId": "sg-014f7b94f96fe2d8d"
}]}
}
}
```

New leader's logs after the controller pods were recycled (10:04–10:06 UTC) — checkpoint-hash skip while AWS had zero rules, then deferred repair 30 minutes later:

```
{"level":"info","ts":"2026-08-28T10:05:36Z","msg":"Skipping targetgroupbinding reconcile","tgb":{"name":"k8s-test-frontend-3208949aed","namespace":"test"},"calculated hash":"iLTpGXt-79uzfeZWF-6nl2HJiDzHRcFtvsksfX60eZ0/9sx_6BNJS1GWy4asYi-EftYturgqZVENv5UBl6sKeKw"}
{"level":"info","ts":"2026-08-28T10:05:36Z","logger":"deferredTGBQueue","msg":"enqueued new deferred TGB","tgb":"k8s-test-frontend-3208949aed"}
... (same for all 8 TGBs)
{"level":"info","ts":"2026-08-28T10:35:48Z","msg":"authorizing securityGroup ingress","securityGroupID":"sg-053bdd55831dccd60","permission":[{"FromPort":3000,...,"ToPort":3000,"UserIdGroupPairs":[{"Description":"elbv2.k8s.aws/targetGroupBinding=shared","GroupId":"sg-014f7b94f96fe2d8d",...}]}]}
{"level":"info","ts":"2026-08-28T10:37:35Z","msg":"authorizing securityGroup ingress",... "ToPort":3008 ...}
{"level":"info","ts":"2026-08-28T10:39:00Z","msg":"authorizing securityGroup ingress",... "ToPort":8025 ...} <- outage ends here
{"level":"info","ts":"2026-08-28T10:48:21Z","msg":"revoking securityGroup ingress",... GC of the now-redundant 3000-3000 and 3000-3008 ...}
```

Full redacted CloudTrail JSON (all 11 controller events) and the complete controller log excerpt are attached: [cloudtrail-events-redacted.json](https://github.com/user-attachments/files/31601745/cloudtrail-events-redacted.json), [controller-log-excerpt-redacted.log](https://github.com/user-attachments/files/31601746/controller-log-excerpt-redacted.log).

**Expected Behavior**

1. A reconcile must never leave the managed permission set on the cluster SG empty while TargetGroupBindings with registered/registering targets exist — regardless of interleaving.
2. The SG info cache should be invalidated (or re-read) whenever a mutation is made to that SG, so a concurrent reconcile cannot diff against pre-mutation state. Today the cache is only bypassed when the first attempt returns an API **error**; a *successful but wrong* revoke escapes detection entirely.
3. After a controller restart, drift between the checkpoint hash's assumption and actual AWS SG state should be repaired promptly, not after the ~30-minute deferred-queue delay.

**Actual Behavior**

- All `elbv2.k8s.aws/targetGroupBinding=shared` rules on the cluster SG were revoked. All ALB→pod traffic (health checks and client traffic) was blocked: `UnHealthyHostCount` = all targets on every target group of all 4 ALBs, clients received HTTP 504.
- The controller itself kept "working" — no error logs, leader healthy — which makes this failure mode invisible from the controller's perspective.
- Duration: ~45 minutes total (09:56–10:39 UTC), of which the last ~30 minutes were purely the deferred-reconcile delay after the controller pods restarted.
- How often: **Rarely** (race condition) — but with total-outage severity. It requires endpoint churn on the TGB that defines the aggregate port-range boundary within the 10-minute cache TTL. A later, much larger pod-churn wave that did *not* touch the boundary-defining TGB produced zero revokes, consistent with the analysis.

**Regression**

No — not a regression. The same symptom was reported against v2.4.1 in #3637 (2024), so the failure class is long-standing. The relevant code paths (`security_group_reconciler.go`, `security_group_manager.go`, `networking_manager.go`) are functionally unchanged from our v3.4.0 through the current v3.5.0 release.

**Current Workarounds**

1. `--disable-restricted-sg-rules=true` — replaces the recomputed port-range rule with an all-ports rule, removing the aggregate-range recomputation that triggers revokes on pod churn (as suggested in #3639). This is what we are adopting.
2. A manually-added static rule on the cluster SG allowing the shared backend SG on the needed port range, with a custom (non-`elbv2.k8s.aws/targetGroupBinding=shared`) description, survives controller garbage collection and acts as a safety net.

**Environment**

- AWS Load Balancer controller version: `v3.4.0` (image `public.ecr.aws/eks/aws-load-balancer-controller:v3.4.0`, GitCommit `3f1e9707ac4ce65504d72b425e82be5245b38e0e`), 2 replicas, leader election on
- Kubernetes version: `v1.36.2-eks-bca9cf6`
- Using EKS (yes/no), if so version?: yes, `v1.36.2-eks-bca9cf6`, **all-Fargate** cluster (target type `ip` everywhere)
- Using Service or Ingress: Ingress (4 ALB ingress groups, 8 TargetGroupBindings across 5 namespaces; backend ports 3000, 3008, 8025)
- AWS region: us-east-1
- How was the aws-load-balancer-controller installed: helm

`helm ls -A | grep -i aws-load-balancer-controller`:
```
aws-load-balancer-controller kube-system 11 2026-07-03 19:54:02 +0200 CEST deployed aws-load-balancer-controller-3.4.0 v3.4.0
```

`helm -n kube-system get values aws-load-balancer-controller`:
```yaml
USER-SUPPLIED VALUES:
clusterName: -cluster
region: us-east-1
serviceAccount:
create: false
name: aws-load-balancer-controller
vpcId: vpc-REDACTED
```

- Current state of the Controller configuration (`kubectl -n kube-system describe deployment aws-load-balancer-controller`, relevant excerpt):
```
Replicas: 2 desired | 2 updated | 2 total | 2 available | 0 unavailable
Image: public.ecr.aws/eks/aws-load-balancer-controller:v3.4.0
Args:
--cluster-name= -cluster
--ingress-class=alb
--aws-region=us-east-1
--aws-vpc-id=vpc-REDACTED
```
(all other flags at defaults — shared backend SG **enabled**, restricted SG rules **enabled**)

- Current state of the Ingress/Service configuration:

`kubectl describe ingressclasses`:
```
Name: alb
Labels: app.kubernetes.io/managed-by=Helm
app.kubernetes.io/version=v3.4.0
helm.sh/chart=aws-load-balancer-controller-3.4.0
Controller: ingress.k8s.aws/alb
```

Representative ingress (`kubectl -n test describe ingress frontend-test-ingress`, one of 4 alike host-based ingress groups; annotations trimmed to the relevant ones):
```
Ingress Class: alb
Rules:
Host Path Backends
app-test.example.com / frontend-alt-service:3000 (:3000)
adm-test.example.com / frontend-adm-service:3000 (:3000)
Annotations:
alb.ingress.kubernetes.io/group.name: -test
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/listen-ports: [{"HTTPS":443}, {"HTTP":80}]
alb.ingress.kubernetes.io/healthcheck-path: /api/healthy
```

Representative service (`kubectl -n test describe svc frontend-alt-service`):
```
Type: ClusterIP
Port: alt 3000/TCP
TargetPort: 3000/TCP
Endpoints: :3000
```

**Possible Solution (Optional)**

Code path at commit `3f1e970` (v3.4.0; unchanged functionally in v3.5.0):

- [`security_group_reconciler.go#L69-L92`](https://github.com/kubernetes-sigs/aws-load-balancer-controller/blob/3f1e9707ac4ce65504d72b425e82be5245b38e0e/pkg/networking/security_group_reconciler.go#L69-L92) — `ReconcileIngress` fetches current rules cache-first, diffs desired vs cached (`diffIPPermissionInfos`, L98/L105), and only retries with `WithReloadIgnoringCache()` if the first attempt **errors**.
- [`security_group_manager.go#L19`](https://github.com/kubernetes-sigs/aws-load-balancer-controller/blob/3f1e9707ac4ce65504d72b425e82be5245b38e0e/pkg/networking/security_group_manager.go#L19) — `defaultSGInfoCacheTTL = 10 * time.Minute`.

Suggested fixes, in rough order of impact:

1. Invalidate (or re-read, ignoring cache) the SG info after every successful authorize/revoke on that SG, so concurrent reconciles cannot diff against pre-mutation state.
2. Sanity-check the net effect of a reconcile: never let the managed permission set end up empty unless the desired set is genuinely empty.
3. Include actual AWS SG rule state (or a post-restart verification pass) in the checkpoint-hash decision, so a controller restart repairs drift immediately instead of after the deferred-queue delay.
4. Serialize reconciles that mutate the same shared SG (per-SG mutex) — the `InvalidPermission.NotFound` races in the CloudTrail trace show multiple workers mutating one SG concurrently.

**Contribution Intention (Optional)**

- [ ] Yes, I'm willing to submit a PR to fix this issue
- [ ] No, I cannot work on a PR at this time

**Additional Context**

Full timeline (2026-08-28, UTC):

| Time | Source | Event |
|---|---|---|
| ~09:52–10:06 | EC2/Fargate | AWS Fargate maintenance recycles pods (ENI churn in CloudTrail); the port-8025 app's pod is replaced |
| 09:55:16–09:56:02 | CloudTrail | The authorize/revoke race described above; zero rules remain after 09:56:01 |
| 09:54→10:39 | CloudWatch | `UnHealthyHostCount` = all targets on every TG of all 4 ALBs; clients receive HTTP 504 |
| 10:04–10:06 | K8s | Both controller pods replaced by the same recycling wave; new leader elected 10:05:36 |
| 10:05:36 | Controller log | New leader skips all 8 TGB reconciles (checkpoint hash match) and defers them — while AWS has zero rules |
| 10:35:48–10:39:00 | Controller log + CloudTrail | Deferred reconciles re-authorize `3000`, `3000-3008`, `3000-8025`; targets healthy again — outage ends |
| 10:48:21 | " | GC revokes the redundant `3000-3000`/`3000-3008`; state converges to a single `3000-8025` rule |

No cluster changes had been made in the ~19 hours before the incident. Related issues: #3637, #3639 (+#3771), #2578.

Attachments: [cloudtrail-events-redacted.json](https://github.com/user-attachments/files/31601745/cloudtrail-events-redacted.json) (all 11 controller CloudTrail events, including the two `InvalidPermission.NotFound` races and the Deregister/RegisterTargets pair proving the pod-replacement trigger), [controller-log-excerpt-redacted.log](https://github.com/user-attachments/files/31601746/controller-log-excerpt-redacted.log) (new leader: startup, lease acquisition, hash-skip lines, deferred repair).

Start with pkg/networking/security_group_manager.go and security_group_reconciler.go, especially the cache TTL and mutation paths cited in the report. Then trace networking_manager.go and the TargetGroupBinding checkpoint and deferred-queue behavior. Done means reproducing the concurrent churn case without losing managed rules and verifying that restart-time AWS drift is repaired promptly.

Written by the indexing model from the issue text.