kubernetes-sigs / kubernetes-sigs/cluster-api

Stale `HealthCheckSucceeded=False` condition not cleared after node recovery when using external remediation

Open
#13,822 13 comments 0 reactions 1 assignee Claimed by @Debasish-87 View on GitHub
area/machinehealthcheck help wanted kind/bug priority/important-soon triage/accepted
Dominant language
Go
Stars
4.3k
Forks
1.6k
Avg merge
1d 3h
Merged PRs (30d)
113

Description

We have encountered an issue where machines end up with a stale `HealthCheckSucceeded=False` condition that never gets cleared, even though the nodes have fully recovered and are running fine.

This has been observed in two scenarios, both using a `MachineHealthCheck` with a `remediationTemplate`:

1. **"No heartbeat for 5 minutes"**: Node briefly reported `Ready=Unknown`, the health check fired, the node recovered on its own, but `HealthCheckSucceeded=False` persists.
2. **"Exceeded start-up time of 30m"**: Node eventually came up healthy after exceeding the configured startup timeout, but the failed condition was never removed.

After a node recovers from such a transient health check failure:

- `NodeHealthy=True`
- `NodeReady=True`
- No active remediation objects exist
- `HealthCheckSucceeded=False`: this was not cleared
- `Available=False`: machine appears unavailable when it is not

**CAPI version used: v1.13.1**

### Label(s) to be applied
/kind bug
/area machinehealthcheck

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.