Generated NetworkPolicies break DNS on GKE clusters using Cloud DNS (link-local resolver cannot be matched by pod selector)
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 608
- Forks
- 263
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 125
Description
Environment
- operator v0.81.1, GKE Standard, Dataplane V2
- cluster created/updated with
--cluster-dns=clouddns --cluster-dns-scope=cluster profile: all(NetworkPolicy defaults enabled)
Symptom
tekton-results-retention-policy-agent crashloops:
lookup tekton-results-postgres-service.tekton-pipelines.svc.cluster.local
on 169.254.169.254:53: read udp ...->169.254.169.254:53: i/o timeout
fatal: Failed to connect to database: timed out waiting for the condition
Root cause
On GKE clusters with Cloud DNS, kubelet writes nameserver 169.254.169.254 (the node-local metadata server running the Cloud DNS data plane) into every pod's resolv.conf. The CoreDNS/kube-dns pods keep running but are unused.
networkpolicy.DNSEgressRule allows DNS egress only to a pod peer (kube-system / k8s-app=kube-dns on port 53). A podSelector/namespaceSelector peer can never match 169.254.169.254 — it is not a pod. Only an ipBlock peer can allow it. On Dataplane V2 the packets are dropped, causing total DNS failure for any pod whose only DNS allow is that rule.
Most generated policies also include APIServerEgressRule() (allow-all, since NP cannot select host-network endpoints), which incidentally permits the link-local resolver and masks the bug. The two policies without it fail visibly:
results-retention-policy-agent(DNS rule + db_port egress only)results-postgres(DNS rule only; latent — postgres does no runtime lookups)
The same applies to NodeLocal DNSCache (169.254.20.10, Autopilot default / opt-in on Standard): it is a host-network DaemonSet unreachable by pod selectors.
Workaround
Override the generated policy by name via spec.networkPolicy.policies (on TektonConfig) and add an ipBlock: 169.254.169.254/32 peer on UDP+TCP 53. Takes effect on next reconcile; no pod restart needed.
Proposed fix
In DNSEgressRule (Kubernetes platform only), add ipBlock peers 169.254.169.254/32 and 169.254.20.10/32 on port 53 alongside the existing kube-dns pod peer. Additive and zero-risk on clusters not using these resolvers — the extra peers simply never match. OpenShift path unchanged.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the Kubernetes platform's DNSEgressRule entry point and inspect how it is used in the generated results-retention-policy-agent and results-postgres policies. Add the two link-local resolver ipBlock peers on UDP and TCP 53 alongside the existing kube-dns peer, then verify the OpenShift path remains unchanged and both affected policies include the new peers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, google-cloud, kubernetes
- Domain
- networking, security
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100