tailscale / tailscale/tailscale
cmd/k8s-operator: ingress ProxyGroup Pods have no readiness gate, so a rolling update can leave a VIPService with zero advertisers
- Dominant language
- Go
- Stars
- 36.5k
- Forks
- 3.2k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 123
Description
Ingress ProxyGroup Pods have no readiness probe or readiness gate, so a StatefulSet rolling update can take every replica out of service at once and leave the VIPService with no advertisers.
This is the TODO at [`cmd/k8s-operator/proxygroup_specs.go#L292-L293`](https://github.com/tailscale/tailscale/blob/d200b3f18f0ee4cff1f6819a78a9fe8e6a7367f2/cmd/k8s-operator/proxygroup_specs.go#L292-L293), present unchanged in v1.98.9 and v1.102.2:
```go
// TODO(tomhjp): add a readiness probe or gate to Ingress Pods. There is a
// small window where the Pod is marked ready but routing can still fail.
```
Filing it because on an HA ingress ProxyGroup that window isn't small, it's an outage.
### Mechanism
`proxyGroupStatefulSet` sets `Replicas` and leaves the rest of the rollout to Kubernetes defaults, so an ingress ProxyGroup StatefulSet comes out as:
- `updateStrategy: RollingUpdate{partition: 0}`
- `podManagementPolicy: OrderedReady`
- `readinessProbe: null`, `livenessProbe: null`, `minReadySeconds: 0`
`RollingUpdate` gates progression from one ordinal to the next on the Pod becoming Ready. With no probe, Ready means "containers started", which is well before tailscaled has authenticated, received a netmap and advertised its VIPService. So the controller deletes replica-1, sees the replacement report Ready almost immediately, and deletes replica-0 while replica-1 is not yet advertising.
For an ingress ProxyGroup fronting a Tailscale Service that leaves the VIP with no advertisers until the new Pod finishes coming up. At the default `replicas: 2` there is no third replica to absorb it, and `minReadySeconds: 0` removes the only other brake. Anything that alters the Pod template will trigger it: a `ProxyClass` env edit, an image bump, an operator upgrade.
### Impact we saw
Two ingress ProxyGroups (`replicas: 2`, operator v1.98.9) fronting an internal Istio gateway exposed as a `LoadBalancer` Service with `ipMode: VIP`. Any ProxyClass change to the group drops the tailnet path to that gateway for the length of the roll, so we now treat routine ProxyGroup edits as outage-risk changes and batch them.
The related egress work doesn't cover this path:
- [#14792](https://github.com/tailscale/tailscale/pull/14792) (merged) and [#20667](https://github.com/tailscale/tailscale/pull/20667) (open) add and attach the `tailscale.com/egress-services` readiness gate, egress only.
- [#14106](https://github.com/tailscale/tailscale/pull/14106) (open) would let users set a `healthz` probe via `ProxyClass`. That checks tailscaled health rather than whether this replica is advertising the Service, so it wouldn't close the window.
Egress also gets a `preStop` hook and a 6-minute deletion grace period. Ingress does shutdown via containerboot's SIGTERM handling instead, for the reason given in the comment above the TODO, but that only covers *un*advertising on the way out, not becoming ready on the way in.
### Suggested fix
A readiness gate for ingress Pods mirroring the egress one: a `tailscale.com/ingress-services` (or similar) readiness gate on the Pod template, with a reconciler that sets the condition only once that replica's advertised services include the VIPServices the group is meant to serve. Defaulting `minReadySeconds` on ingress ProxyGroups would help on its own.
### Environment
- operator v1.98.9; the TODO is unchanged in v1.102.2 and `main`
- ProxyGroup `type: ingress`, `replicas: 2`, consumer is a `LoadBalancer` Service with `tailscale.com/proxy-group`, `ipMode: VIP`
- Kubernetes v1.35.4 (kubespray, self-managed)
Contributor guide
Research direction
Start at cmd/k8s-operator/proxygroup_specs.go#L292-L293 and inspect proxyGroupStatefulSet, then compare the related egress readiness-gate work in PRs #14792 and #20667. Define completion as ingress rollouts waiting until each replacement Pod is advertising its VIPServices, without leaving the VIPService with zero advertisers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100