kubernetes-sigs / kubernetes-sigs/node-readiness-controller

[BUG] Handle node deletion, label selector unmatching, and status drift in NodeReconciler

Open
#338 7 comments 0 reactions 1 assignee View on GitHub

@bhuvan-somisetty is already working on this.

Since Aug 2, 2026.

Dominant language
Go
Stars
164
Forks
75
Avg merge
8d 23h
Merged PRs (30d)
13

Description

What happened?

When a node is deleted from the cluster, NodeReconciler.Reconcile gets a NotFound error from r.Get and silently swallows it via client.IgnoreNotFound(err). The deleted node is never removed from rule.Status.NodeEvaluations, AppliedNodes, or FailedNodes on any rule, and the node_readiness_nodes_by_state metric keeps counting it forever.

Separately, RuleReconciler.cleanupDeletedNodes only filters NodeEvaluations — it misses AppliedNodes and FailedNodes, so those two fields also go stale.

Two related points originally filed here are split out per this review:

  • AppliedNodes real-time sync — deferred until #89 lands (per-node status is moving out of rule.status entirely).
  • Orphaned taints on selector unmatch — moved to #342 (needs its own design discussion).
Expected behavior
  • Deleting a node removes it from NodeEvaluations, AppliedNodes, and FailedNodes on every rule, and updates the metric.
  • cleanupDeletedNodes cleans up all three fields, not just NodeEvaluations.
Repro
  1. Create a NodeReadinessRule matching worker nodes.
  2. Confirm a node shows up in rule.Status.NodeEvaluations/AppliedNodes.
  3. kubectl delete node worker-1.
  4. worker-1 is still in the rule status and in /metrics.
Affected components
  • internal/controller/node_controller.go (Reconcile)
  • internal/controller/nodereadinessrule_controller.go (cleanupDeletedNodes)
Controller version / K8s version

main (v0.4.1+) / v1.30.0v1.31.0

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.