[FEATURE] [LFX] Node Readiness SLO Dashboard & Operational Metrics

1. Problem Statement

Node Readiness Controller (NRC) prevents pods from landing on uninitialized nodes via taints, operators lack a visual "single pane of glass" to answer:

• Where is the bottleneck? Which specific initialization rule is slowing down scale-up?
• Is it reliable? Are driver initializations failing or flapping?
• What is the cost? How much time are nodes spending in a "Tainted" state before workloads can start?

---

2. Proposed Solution

We propose creating a standardized Operational Dashboard (pluggable for Grafana, Headlamp, or other UI tools) that surfaces NRC Prometheus metrics and maps them to actionable Service Level Objectives (SLOs).

---

3. Metrics & SLO Framework

The dashboard will be centered around four key reliability pillars but not limited to:

SLO Category	Target Goal	Primary Metric	Purpose
Node Bootstrap Latency	95% < 5m	node_readiness_bootstrap_completed_total	Tracks "Time-to-Ready" lag for new nodes.
Controller Performance	99% < 50ms	node_readiness_evaluation_duration_seconds	Ensures NRC isn't a bottleneck during mass joins.
Infrastructure Stability	99.9% Success	node_readiness_failures_total	Detects flaky drivers, CNI, or DRA providers.
Operational Churn	N/A	node_readiness_taint_operations_total	Monitors the volume of taint additions vs. removals.

---

4. Key Visualizations

A. The Bottleneck Heatmap

A view that ranks NodeReadinessRule objects by the average time it takes for their conditions to be satisfied. This allows SREs to pinpoint exactly which infrastructure component (e.g., DRA, local storage, CNI) is stalling cluster expansion.

B. Time-to-Ready Distribution

A timeline chart showing the delta between Node.CreationTimestamp and the removal of the final NRC taint. This represents the "wasted" time where hardware is powered on but idle(maybe we can expose metrics specific to this).

C. Active Blocker List

A real-time table of nodes currently held by NRC, filtered by:

• Node Name
• Pending Condition (e.g., DRADriverReady=False)
• Time in State (to identify "stuck" nodes)

D. Error Budget Tracking

A high-level health indicator visualizing the Rule Compliance Ratio. If the failure rate of initialization rules exceeds 0.1%, it triggers a visual warning that the underlying infrastructure is unstable.

---

We should be able to publish the dashboard on headlamp(https://github.com/headlamp-k8s/plugins).

[ajaysundark] Update: This project is now proposed for LFX mentorship program. Please refrain from self-assigning yourself this feature.