Azure / Azure/AKS

[BUG] Istio add-on upgrade leaves orphaned leader-election Leases in aks-istio-system

Open
#5,862 3 comments 0 reactions 1 assignee Claimed by @therealmitchconnors View on GitHub
action-required bug istio networking stale
Dominant language
TypeScript
Stars
2.1k
Forks
395
Avg merge
2d 22h
Merged PRs (30d)
13

Description

**Describe the bug**
After upgrading the AKS Istio add-on from one revision to another and completing the canary (old revision fully retired), Coordination v1 `Lease` objects created by the retired revision's istiod remain in the `aks-istio-system` namespace indefinitely. They are never renewed by any running pod, so `kube_lease_renew_time` grows monotonically stale.

Old-revision Lease objects persist indefinitely after upgrade completion. `holderIdentity` on each still points at a pod that no longer exists. `renewTime` is frozen at the moment the old istiod was terminated.

This is not limited to a single test cluster. On production AKS clusters that have gone through multiple Istio add-on upgrades, orphan leases accumulate from every retired revision:

- `istio-gateway-deployment-asm-1-25` — revision retired ~60+ days ago
- `istio-gateway-deployment-asm-1-26` — revision retired ~60 days ago
- `istio-gateway-status-leader-asm-1-26` — same

This means any AKS cluster that has ever completed an Istio add-on upgrade will have orphan leases from every previously retired revision, and the count grows with each subsequent upgrade.

These leases persist across multiple clusters in different Azure regions, confirming this is systemic add-on behaviour, not environment-specific.

Graphing `time() - kube_lease_renew_time{namespace="aks-istio-system"}` shows a linear climb for each orphan lease — consistent with a Lease object in etcd that nothing is renewing.

**To Reproduce**
Tested 2026-07-14 on a fresh AKS cluster. Total time: ~20 minutes.

**Step 1 — Create AKS cluster with Istio add-on at asm-1-28**

```bash
az aks create -g -n \
--node-count 2 --node-vm-size Standard_D2s_v3 \
--enable-asm --revision asm-1-28 \
--generate-ssh-keys

az aks mesh enable-ingress-gateway -g -n \
--ingress-gateway-type external
```

**Step 2 — Baseline: only asm-1-28 leases exist, actively renewed**

```
$ kubectl -n aks-istio-system get leases -o wide
NAME HOLDER AGE
istio-gateway-deployment-asm-1-28 istiod-asm-1-28-758f6d9fb-gpsqj 32m
istio-gateway-status-leader-asm-1-28 istiod-asm-1-28-758f6d9fb-gpsqj 32m
```

Both leases are actively renewed by the running istiod pod.

**Step 3 — Canary upgrade to asm-1-29 and complete**

```bash
az aks mesh upgrade start -g -n --revision asm-1-29
# Wait ~3-5 min for new istiod to come up
az aks mesh upgrade complete -g -n
# Wait ~2-3 min for asm-1-28 pods to terminate
```

**Step 4 — Verify only asm-1-29 pods are running**

```
$ kubectl -n aks-istio-system get pods
NAME READY STATUS RESTARTS AGE
istiod-asm-1-29-fb954fd5b-r2jdc 1/1 Running 0 14m
istiod-asm-1-29-fb954fd5b-sb8rk 1/1 Running 0 13m
```

No `asm-1-28` pods exist — the old revision has been fully retired.

**Step 5 — BUG: asm-1-28 leases still present**

```
$ kubectl -n aks-istio-system get leases -o wide
NAME HOLDER AGE
istio-gateway-deployment-asm-1-28 istiod-asm-1-28-758f6d9fb-gpsqj 56m <-- ORPHAN
istio-gateway-deployment-asm-1-29 istiod-asm-1-29-fb954fd5b-r2jdc 14m
istio-gateway-status-leader-asm-1-28 istiod-asm-1-28-758f6d9fb-gpsqj 56m <-- ORPHAN
istio-gateway-status-leader-asm-1-29 istiod-asm-1-29-fb954fd5b-r2jdc 14m
```

The `*-asm-1-28` leases reference `istiod-asm-1-28-758f6d9fb-gpsqj` — a pod that no longer exists (see Step 4).

**Step 6 — Orphan lease YAML confirms frozen renewTime**

```yaml
apiVersion: coordination.k8s.io/v1
kind: Lease
metadata:
creationTimestamp: "2026-07-14T01:48:53Z"
name: istio-gateway-deployment-asm-1-28
namespace: aks-istio-system
spec:
acquireTime: "2026-07-14T01:48:52.915577Z"
holderIdentity: istiod-asm-1-28-758f6d9fb-gpsqj
leaseDurationSeconds: 30
leaseTransitions: 0
renewTime: "2026-07-14T02:36:28.971430Z"
```

`renewTime` is frozen at `02:36:28Z` — the moment the asm-1-28 istiod was terminated. It will never be updated again because no pod with that identity exists.

By contrast, the active `asm-1-29` lease's `renewTime` updates every ~5 seconds.

**Expected behavior**
When a revision is retired and its Deployments/ReplicaSets are scaled down as part of `az aks mesh upgrade complete`, the Lease objects owned by that revision's istiod should be garbage-collected — either via an ownerReference on the Lease pointing at the revision's Deployment, or via explicit cleanup in the add-on's upgrade controller.

**Screenshots**
If applicable, add screenshots to help explain your problem. Make sure not to include sensitive or personal information.

**Environment (please complete the following information):**

- AKS clusters with the Istio service mesh add-on enabled
- Reproduced on a fresh AKS cluster (2-node, Standard_D2s_v3) in ``
- Also observed across multiple production AKS clusters in different Azure regions
- AKS Kubernetes version: 1.35
- Istio add-on revisions tested: `asm-1-28` (v1.28.8-2) → `asm-1-29` (v1.29.4-1)
- Detected via `kube_lease_renew_time` from kube-state-metrics

**Additional context**
**Impact:**

- **False-positive alerts:** Any fleet-wide leader-election health monitoring (e.g. `kube_lease_renew_time` staleness alerts) fires indefinitely on these orphan leases, requiring manual exclusion rules.
- **Unbounded etcd accumulation:** Each revision upgrade adds ~2 orphan leases per cluster. Small individually, but unbounded over the product lifetime across a fleet.
- **Diagnostic noise:** During incident triage, orphan leases make it harder to distinguish genuinely broken leader elections from leftover noise.

**Suggested fix:**

Either:
1. **ownerReference:** Set an `ownerReference` on each Lease pointing at the owning revision's Deployment, so Kubernetes GC removes the Lease when the Deployment is deleted/scaled to zero, **or**
2. **Explicit cleanup:** Add a cleanup step in the add-on's `upgrade complete` path that deletes Leases in `aks-istio-system` whose name suffix matches the retired revision (e.g. `*-asm-1-28`).

**Reproduction summary:**

| Step | Command | Expected | Actual |
|------|---------|----------|--------|
| Install asm-1-28 | `az aks create --enable-asm --revision asm-1-28` | 2 leases created | 2 leases created |
| Enable ingress | `az aks mesh enable-ingress-gateway` | Gateway leases appear | Gateway leases appear |
| Upgrade to asm-1-29 | `az aks mesh upgrade start/complete` | Old leases removed | **Old leases persist** |
| Verify pods | `kubectl get pods` | Only asm-1-29 pods | Only asm-1-29 pods |
| Verify leases | `kubectl get leases` | Only asm-1-29 leases | **Both asm-1-28 and asm-1-29 leases** |

Happy to test a fix on our clusters if useful.

## Additional ARO HCP validation

This issue is also reproducible in ARO HCP production service clusters:

- `prod-centralindia-svc-1`
- `prod-brazilsouth-svc-1`
- Namespace: `aks-istio-system`

Observed leases:

| Lease | Status |
|---|---|
| `istio-gateway-deployment-asm-1-28` | Active and renewing |
| `istio-gateway-status-leader-asm-1-28` | Active and renewing |
| `istio-gateway-deployment-asm-1-26` | Orphaned; not renewed |
| `istio-gateway-status-leader-asm-1-26` | Orphaned; not renewed |
| `istio-gateway-deployment-asm-1-25` | Orphaned; not renewed |

The old leases have frozen `renewTime` values, while the active `asm-1-28` leases continue renewing normally. This confirms that the issue is caused by retired ASM revisions leaving their leader-election Lease objects behind after their workloads are removed.

This is currently an alerting/operational-noise issue rather than an Istio gateway outage. However, it will recur on every revision upgrade: when the environment moves from `asm-1-28` to `asm-1-29`, the `asm-1-28` leases are expected to become orphaned in the same way unless the revision-retirement process explicitly removes them.

### Expected fix

The Istio add-on revision-retirement workflow should delete the revision-specific `istio-gateway-*` Lease objects when the corresponding ASM revision is removed. Deleting the stale leases manually is only a short-term workaround and does not prevent recurrence.

Please confirm whether the AKS Istio add-on team can add lease cleanup to revision retirement, or otherwise ensure that these leases receive lifecycle ownership/garbage collection.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.