kubernetes-sigs / kubernetes-sigs/node-readiness-controller

[BUG] Controller emits zero Warning events on NodeReadinessRule or Node objects when taint operations fail

Open
#362 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

kind/bug triage/needs-information
Dominant language
Go
Stars
164
Forks
75
Avg merge
8d 23h
Merged PRs (30d)
13

Description

What happened?

The controller emits Kubernetes Events for successful taint operations on Node objects, but emits no Warning events when reconciliation or taint operations fail. Additionally, no events are ever emitted on the owning NodeReadinessRule object.

When addTaintBySpec or removeTaintBySpec fails (for example due to API server conflicts, RBAC restrictions, or node patch failures), or when node evaluation fails in processAllNodesForRule / processNodeAgainstAllRules, the controller logs the error and increments metrics.Failures, but never calls EventRecorder.Eventf on either the NodeReadinessRule or the affected Node.

As a result:

  • kubectl describe nodereadinessrule <rule> shows:
    Events: <none>
    
  • kubectl describe node <node> only shows historical successful events (such as TaintAdded or TaintRemoved) and provides no indication of reconciliation failures.
  • Operators must rely on controller logs or Prometheus metrics to diagnose scheduling or reconciliation issues.
Steps to Reproduce
  1. Create a NodeReadinessRule with:
    enforcementMode: continuous
    
  2. Induce a failure during addTaintBySpec or removeTaintBySpec (for example by restricting RBAC permissions for patching nodes or forcing an API patch conflict).
  3. Observe that the controller logs an error and increments metrics.Failures.
  4. Run:
    kubectl describe nodereadinessrule <rule>
    
    Observe:
    Events: <none>
    
  5. Run:
    kubectl get events --field-selector type=Warning
    
    Observe that no Warning event is recorded for the failed reconciliation.
Expected Behavior

When node evaluation or taint operations fail, the controller should emit Kubernetes Warning events following standard controller conventions.

Specifically:

  • Emit a Warning event on the owning NodeReadinessRule describing the reconciliation failure.
  • Emit a corresponding Warning event on the affected Node when taint application or removal fails.
  • Use meaningful event reasons such as AddTaintError, RemoveTaintError, or EvaluationError so operators can diagnose failures directly through kubectl describe.
Controller Version / Image Tag

v0.4.1

Kubernetes Version

v1.36.0 (reproduced on both envtest and kind)

Controller Logs
2026-08-06T09:40:00Z ERROR Failed to evaluate node for rule {"rule":"network-ready","node":"worker-1","error":"failed to remove taint: node \"worker-1\" cannot be patched"}
Additional Environment Details

NIL

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating processAllNodesForRule, processNodeAgainstAllRules, addTaintBySpec, removeTaintBySpec, and existing EventRecorder.Eventf calls. Reproduce the failure with envtest or kind, then verify that failed evaluation emits a Warning on the NodeReadinessRule and failed taint operations also emit one on the affected Node with the requested reasons.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.