tailscale / tailscale/tailscale
k8s-operator: DNSConfig nameserver never provisions on OpenShift — ClusterRole missing dnsconfigs/finalizers
- Dominant language
- Go
- Stars
- 36.5k
- Forks
- 3.2k
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 132
Description
## What happened
On OpenShift, applying a `DNSConfig` never provisions a nameserver. The
`nameserver-reconciler` fails on every attempt and retries forever with exponential
backoff. The `DNSConfig` gets no `status`, so `dns-records-reconciler` sits at
`DNSConfig is not ready yet, waiting...` for every proxy indefinitely.
```
{"level":"error","logger":"nameserver-reconciler","controllerKind":"DNSConfig",
"DNSConfig":{"name":"ts-dns"},
"error":"error provisioning nameserver resources: error reconciling ServiceAccount:
serviceaccounts \"nameserver\" is forbidden: cannot set blockOwnerDeletion if an
ownerReference refers to a resource you can't set finalizers on: , "}
```
## Why
OpenShift enables the [`OwnerReferencesPermissionEnforcement`](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#ownerreferencespermissionenforcement)
admission plugin **by default**; upstream Kubernetes does not. That plugin requires the
creating identity to hold `update` on the **owner's** `finalizers` subresource before it
will accept a child carrying `blockOwnerDeletion: true`.
The nameserver's children (ServiceAccount, Deployment, Service, ConfigMap) are created
with an ownerReference to the **cluster-scoped** `DNSConfig` and `blockOwnerDeletion: true`.
The operator's ClusterRole grants `dnsconfigs` and `dnsconfigs/status`, but no rule on any
`finalizers` subresource:
https://github.com/tailscale/tailscale/blob/main/cmd/k8s-operator/deploy/chart/templates/operator-rbac.yaml
```yaml
- apiGroups: ["tailscale.com"]
resources: ["dnsconfigs", "dnsconfigs/status"]
verbs: ["get", "list", "watch", "update"]
```
So the admission plugin rejects the ServiceAccount create, and the reconciler can never
make progress. The whole chart contains **no `*/finalizers` rule at all** — confirmed at
`v1.98.4`, at `v1.102.3`, and on `main` at the time of filing.
## Reproduce
1. OpenShift (any recent 4.x — `OwnerReferencesPermissionEnforcement` is on by default).
2. Install the operator chart.
3. `kubectl apply` a minimal `DNSConfig`:
```yaml
apiVersion: tailscale.com/v1alpha1
kind: DNSConfig
metadata:
name: ts-dns
spec:
nameserver:
image:
repo: tailscale/k8s-nameserver
tag: v1.98.4
```
4. `kubectl logs deploy/operator -n tailscale` — the error above, looping.
`kubectl get dnsconfig ts-dns -o yaml` — no `status` block.
## Fix
Add to the ClusterRole in `cmd/k8s-operator/deploy/chart/templates/operator-rbac.yaml`:
```yaml
- apiGroups: ["tailscale.com"]
resources: ["dnsconfigs/finalizers"]
verbs: ["update"]
```
Verified: with exactly that rule granted, the reconciler succeeds on its next attempt and
the nameserver comes up normally (`NameserverReady: True`, `status.nameserver.ip` populated,
`dnsrecords` ConfigMap populated from EndpointSlices). No other change was needed, and the
grant is inert on clusters where the admission plugin is off.
## Possibly broader than DNSConfig
I only hit and verified this for `DNSConfig`, because that is the CR whose children are
owned by a cluster-scoped resource in my setup. The same admission rule applies to any
owner/child pair the operator creates with `blockOwnerDeletion: true`, so `Connector`,
`ProxyGroup`, `ProxyGroupPolicy` and `Recorder` may need the equivalent rule — worth a look
from someone who knows which of those set `blockOwnerDeletion`.
## Workaround
An additive ClusterRole + ClusterRoleBinding granting the rule to the operator's
ServiceAccount, rather than patching the chart's own ClusterRole (which is then free to be
managed by the chart):
```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: tailscale-operator-dnsconfig-finalizers
rules:
- apiGroups: ["tailscale.com"]
resources: ["dnsconfigs/finalizers"]
verbs: ["update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: tailscale-operator-dnsconfig-finalizers
subjects:
- kind: ServiceAccount
name: operator
namespace: tailscale
roleRef:
kind: ClusterRole
name: tailscale-operator-dnsconfig-finalizers
apiGroup: rbac.authorization.k8s.io
```
## Environment
- Operator `docker.io/tailscale/k8s-operator:v1.98.4`, nameserver `tailscale/k8s-nameserver:v1.98.4`
- OpenShift 4.x
- Reproduced on a single-node cluster; nothing about it looks node-count dependent
Happy to send a PR for the one-rule change if that is useful.
Contributor guide
Research direction
Start in cmd/k8s-operator/deploy/chart/templates/operator-rbac.yaml and inspect the operator ClusterRole rules for DNSConfig. Add the requested finalizers permission, then render or validate the chart and confirm that DNSConfig reconciliation reaches NameserverReady with the nameserver status populated on OpenShift.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- helm, kubernetes
- Domain
- infrastructure
- Issue type
- Bug
- Difficulty
- 1/5
- Estimated time
- Under an hour
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 88/100