Agent-Hellboy / Agent-Hellboy/mcp-runtime
Harden platform recovery for cluster resource pressure and workload eviction events
- Ngôn ngữ chính
- Go
- Star
- 6
- Fork
- 1
- Merge trung bình
- 11 giờ 33 phút
- Pull request đã merge (30 ngày)
- 13
Mô tả
The platform should recover cleanly from generic cluster resource-pressure events without requiring manual
intervention.
We observed a failure pattern where workload evictions and control-plane churn left parts of the platform
unavailable even after underlying node capacity was restored. Recovery required manual cleanup of stale pods,
restarting cluster services, and resetting specific stateful components.
This is not a cluster-vendor-specific problem. The platform should behave predictably on any Kubernetes
distribution when nodes experience disk pressure, image pull instability, or pod eviction storms.
Problem
Current recovery behavior has a few gaps:
- stale Completed, Error, and ContainerStatusUnknown pods accumulate after eviction churn
- internal dependencies do not always recover in the right order
- readiness can remain blocked after dependencies come back
- image pull failures can cascade into prolonged service unavailability
- some stateful components can become unrecoverable without manual reset when backing dependencies are recreated
Goal
Make the platform resilient to generic Kubernetes resource-pressure and eviction scenarios, with deterministic
recovery and minimal operator intervention.
Scope
- audit sentinel and core platform workloads for startup, readiness, and dependency recovery behavior
- ensure services recover automatically when dependencies return after transient failure
- reduce sensitivity to transient image-pull failures and registry instability
- define safe handling for stale pods left behind after eviction or restart churn
- improve stateful workload recovery semantics when dependent control-plane services are recreated
- add preflight or runtime checks for low-resource conditions that are likely to cause cascading failures
- add an operator repair path for partial platform recovery that does not require manual pod/PVC surgery
Acceptance criteria
- after node/resource pressure is relieved, platform workloads converge back to healthy state automatically or
via a single documented repair action
- transient image pull failures do not leave long-lived control-plane services stuck indefinitely
- services depending on messaging/storage recover once those dependencies are healthy
- stale failed/completed pods do not obscure real platform state or require manual cleanup during normal recovery
- stateful components have a documented and supported recovery strategy for dependency re-creation scenarios
- add automated coverage or a reproducible recovery test for eviction/resource-pressure scenarios
Notes
This issue is about platform-level recovery hardening, not vendor-specific tuning for k3s, EKS, GKE, AKS, or
kind. The fix should apply generically to Kubernetes clusters.
Hướng dẫn đóng góp
Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này
Đánh giá
Issue này chưa được đánh giá.