digitalocean / digitalocean/digitalocean-cloud-controller-manager
Duplicate do-loadbalancer-name across clusters destroys existing LB and permanently loses IP
- Dominant language
- Go
- Stars
- 574
- Forks
- 153
- Avg merge
- 3d 6h
- Merged PRs (30d)
- 3
Description
## Description
When deploying a second DOKS cluster with an `ingress-nginx` service using the same `service.beta.kubernetes.io/do-loadbalancer-name` annotation as an existing LB on a different cluster, the **existing LB was destroyed and replaced** with a new one in a different region. The original IP was permanently lost, causing a full production outage.
This is distinct from #556 where the LB "moved" between clusters but retained the same IP. In our case, the LB was deleted and recreated, resulting in **permanent IP loss**.
## Steps to reproduce
1. Create DOKS cluster **A** in `nyc3`
2. Deploy a `LoadBalancer` service with annotation `do-loadbalancer-name: "my-lb"` → LB created with IP `X.X.X.X` in nyc3
3. Create DOKS cluster **B** in `fra1`
4. Deploy a `LoadBalancer` service with the **same** annotation `do-loadbalancer-name: "my-lb"`
5. The LB from cluster A is **destroyed**. A new LB is created in `fra1` with a **different IP**
6. Cluster A's service still shows the old IP in `EXTERNAL-IP` but the LB no longer exists
7. All traffic to the old IP is permanently lost
## Expected behavior
Step 4 should **fail** with an error such as:
- "Load balancer name already in use"
- "Quota exceeded (2/2)"
The existing LB should **never** be silently destroyed.
## Actual behavior
- The existing production LB was destroyed without any warning or error
- A new LB was created in a different region (`fra1` instead of `nyc3`) with a new IP
- The old IP was permanently lost (DO does not support reserved IPs for LBs)
- The LB quota (2/2) was not enforced — instead of rejecting the creation, the backend replaced the existing LB
- Cluster A's service was left in a stale state, showing a dead `EXTERNAL-IP`
## Impact
- Full production outage across all services behind the LB
- Permanent loss of the original IP address
- All Cloudflare DNS records had to be manually updated to the new IP
- Required manual firewall reconfiguration and LB forwarding rule fixes
## Environment
- **DOKS** (DigitalOcean Managed Kubernetes)
- Two clusters in different regions (nyc3 and fra1)
- CCM managed by DOKS control plane (not self-hosted)
- No `kubernetes.digitalocean.com/load-balancer-id` annotation was set — only `do-loadbalancer-name`
## Key difference from #556
In #556, the LB ownership "moved" between clusters but the LB itself (and its IP) was preserved. In our case, the LB was **destroyed and recreated in a different region**, causing irreversible IP loss. This suggests the behavior may differ when clusters are in different regions — instead of reassigning the existing LB, the backend deletes it and creates a new one.
## Suggested fix
1. The API/CCM should **reject** creating a LB with a name that already exists (unless explicit ownership transfer via `load-balancer-id` annotation)
2. At minimum, the quota check should prevent this — if the account is at 2/2 LBs, creating a new one should fail, not replace an existing one
3. Cross-region name conflicts should be treated as errors, not as implicit delete-and-recreate
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.