Agent-Hellboy / Agent-Hellboy/mcp-runtime

Harden platform recovery for cluster resource pressure and workload eviction events

未关闭
#72 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Go
星标
6
派生
1
平均合并
11 小时 33 分钟
30 天内合并 PR
13

描述

The platform should recover cleanly from generic cluster resource-pressure events without requiring manual
intervention.

We observed a failure pattern where workload evictions and control-plane churn left parts of the platform
unavailable even after underlying node capacity was restored. Recovery required manual cleanup of stale pods,
restarting cluster services, and resetting specific stateful components.

This is not a cluster-vendor-specific problem. The platform should behave predictably on any Kubernetes
distribution when nodes experience disk pressure, image pull instability, or pod eviction storms.

Problem
Current recovery behavior has a few gaps:

- stale Completed, Error, and ContainerStatusUnknown pods accumulate after eviction churn
- internal dependencies do not always recover in the right order
- readiness can remain blocked after dependencies come back
- image pull failures can cascade into prolonged service unavailability
- some stateful components can become unrecoverable without manual reset when backing dependencies are recreated

Goal
Make the platform resilient to generic Kubernetes resource-pressure and eviction scenarios, with deterministic
recovery and minimal operator intervention.

Scope

- audit sentinel and core platform workloads for startup, readiness, and dependency recovery behavior
- ensure services recover automatically when dependencies return after transient failure
- reduce sensitivity to transient image-pull failures and registry instability
- define safe handling for stale pods left behind after eviction or restart churn
- improve stateful workload recovery semantics when dependent control-plane services are recreated
- add preflight or runtime checks for low-resource conditions that are likely to cause cascading failures
- add an operator repair path for partial platform recovery that does not require manual pod/PVC surgery

Acceptance criteria

- after node/resource pressure is relieved, platform workloads converge back to healthy state automatically or
via a single documented repair action
- transient image pull failures do not leave long-lived control-plane services stuck indefinitely
- services depending on messaging/storage recover once those dependencies are healthy
- stale failed/completed pods do not obscure real platform state or require manual cleanup during normal recovery
- stateful components have a documented and supported recovery strategy for dependency re-creation scenarios
- add automated coverage or a reproducible recovery test for eviction/resource-pressure scenarios

Notes
This issue is about platform-level recovery hardening, not vendor-specific tuning for k3s, EKS, GKE, AKS, or
kind. The fix should apply generically to Kubernetes clusters.

贡献指南

这个仓库没有索引到贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。