kubernetes-sigs / kubernetes-sigs/cluster-api
Stale `HealthCheckSucceeded=False` condition not cleared after node recovery when using external remediation
- Dominant language
- Go
- Stars
- 4.3k
- Forks
- 1.6k
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 113
Description
We have encountered an issue where machines end up with a stale `HealthCheckSucceeded=False` condition that never gets cleared, even though the nodes have fully recovered and are running fine.
This has been observed in two scenarios, both using a `MachineHealthCheck` with a `remediationTemplate`:
1. **"No heartbeat for 5 minutes"**: Node briefly reported `Ready=Unknown`, the health check fired, the node recovered on its own, but `HealthCheckSucceeded=False` persists.
2. **"Exceeded start-up time of 30m"**: Node eventually came up healthy after exceeding the configured startup timeout, but the failed condition was never removed.
After a node recovers from such a transient health check failure:
- `NodeHealthy=True`
- `NodeReady=True`
- No active remediation objects exist
- `HealthCheckSucceeded=False`: this was not cleared
- `Available=False`: machine appears unavailable when it is not
**CAPI version used: v1.13.1**
### Label(s) to be applied
/kind bug
/area machinehealthcheck
Contributor guide
Assessment
This issue has not been assessed yet.