Agent-Hellboy / Agent-Hellboy/mcp-runtime

Harden platform recovery for cluster resource pressure and workload eviction events

Đang mở
#72 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Go
Star
6
Fork
1
Merge trung bình
11 giờ 33 phút
Pull request đã merge (30 ngày)
13

Mô tả

The platform should recover cleanly from generic cluster resource-pressure events without requiring manual
intervention.

We observed a failure pattern where workload evictions and control-plane churn left parts of the platform
unavailable even after underlying node capacity was restored. Recovery required manual cleanup of stale pods,
restarting cluster services, and resetting specific stateful components.

This is not a cluster-vendor-specific problem. The platform should behave predictably on any Kubernetes
distribution when nodes experience disk pressure, image pull instability, or pod eviction storms.

Problem
Current recovery behavior has a few gaps:

- stale Completed, Error, and ContainerStatusUnknown pods accumulate after eviction churn
- internal dependencies do not always recover in the right order
- readiness can remain blocked after dependencies come back
- image pull failures can cascade into prolonged service unavailability
- some stateful components can become unrecoverable without manual reset when backing dependencies are recreated

Goal
Make the platform resilient to generic Kubernetes resource-pressure and eviction scenarios, with deterministic
recovery and minimal operator intervention.

Scope

- audit sentinel and core platform workloads for startup, readiness, and dependency recovery behavior
- ensure services recover automatically when dependencies return after transient failure
- reduce sensitivity to transient image-pull failures and registry instability
- define safe handling for stale pods left behind after eviction or restart churn
- improve stateful workload recovery semantics when dependent control-plane services are recreated
- add preflight or runtime checks for low-resource conditions that are likely to cause cascading failures
- add an operator repair path for partial platform recovery that does not require manual pod/PVC surgery

Acceptance criteria

- after node/resource pressure is relieved, platform workloads converge back to healthy state automatically or
via a single documented repair action
- transient image pull failures do not leave long-lived control-plane services stuck indefinitely
- services depending on messaging/storage recover once those dependencies are healthy
- stale failed/completed pods do not obscure real platform state or require manual cleanup during normal recovery
- stateful components have a documented and supported recovery strategy for dependency re-creation scenarios
- add automated coverage or a reproducible recovery test for eviction/resource-pressure scenarios

Notes
This issue is about platform-level recovery hardening, not vendor-specific tuning for k3s, EKS, GKE, AKS, or
kind. The fix should apply generically to Kubernetes clusters.

Hướng dẫn đóng góp

Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.