Planner does not re-fire `apply-statefulset` when StatefulSet is deleted post-bootstrap
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 58/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Quiet
- Tech stack
- go, kubernetes
- Domain
- devops, infrastructure
Research direction
Start with the SeiNode reconciler and planner path described in the issue, focusing on the post-bootstrap phase: Running handling and the apply-statefulset, apply-service, and apply-rbac-proxy-config tasks. Trace how derived resources are currently checked and how chain-upgrade and peer re-discovery plans are triggered. Done means a missing StatefulSet is recreated from current SeiNode.Spec without rerunning bootstrap work or deleting the data PVC, while existing reconcile paths remain unchanged.
Written by the indexing model from the issue text.
Description
Problem
When a SeiNode reaches phase: Running, the controller's planner is finished. If the rendered StatefulSet is later deleted (manually for ops reasons, or as part of an aftercare sweep), the SeiNode reconciler does not detect the missing derived resource and re-fire the apply-statefulset task. The SeiNode stays in phase: Running indefinitely with no live pod.
Live-reproduced today against the state-size-analyzer SND in pacific-1: deleted the StatefulSet to force a sidecar-image re-render via the platform-default SEI_SIDECAR_IMAGE env. Controller never recreated it. Workaround required deleting the SeiNode itself so the SND's reconcileSeiNodes would recreate it fresh and the bootstrap plan would re-run from scratch — which also wipes the data PVC (SeiNode-owned), forcing a full state-sync redo.
Impact
Any ops procedure that deletes the StatefulSet becomes a one-way door — the SND won't bring seid back automatically. The most immediate consumer is the state-size-analysis CronJob (queued follow-up to platform state-size-analyzer.yaml) which is designed to scale replicas: 0, run an analyzer Job against the released PVC, and scale back — that pattern's correctness depends on the controller re-creating the StatefulSet on the scale-up path. Today operators have to know the "delete the SeiNode and accept PVC loss" workaround, which is a footgun: silent state loss for anyone who doesn't realize the cascade.
Relevant experts
kubernetes-specialist— controller planner + reconcile logic
Proposed approach
In the SeiNode reconciler, after the bootstrap plan completes (phase: Running), continue to assert derived resources exist on each reconcile pass. If the StatefulSet matching SeiNode.Name is missing, fire a new plan containing only the post-bootstrap apply tasks (apply-statefulset, apply-service, apply-rbac-proxy-config if TLS is enabled). Do not re-fire discover-peers / configure-state-sync / config-validate — those already ran and the existing PVC carries their result. The new plan should be a targeted "rebuild-derived-resources" flow, not a full bootstrap.
Acceptance criteria
- Deleting the rendered StatefulSet on a Running SeiNode causes the controller to recreate it within one reconcile cycle
- The new StatefulSet is rendered from current
SeiNode.Spec(so any sidecar-image change since initial bootstrap is picked up via the platform default) - The data PVC is preserved across the recreate — no state-sync redo
- Existing reconcile paths (chain-upgrade plan, peer re-discovery) remain unaffected
Out of scope
The related "ensure-data-pvc fails terminally when a stale orphaned PVC is present from a just-deleted SeiNode" race condition. That's a separate issue — that code path needs either auto-adoption logic or retry-with-backoff so K8s GC has time to catch up on the orphan. Filing separately if/when it comes up again.
References
- Live reproduction today on the
pacific-1/state-size-analyzerSND - Triggering context: cycling the pod to pick up the platform-default
SEI_SIDECAR_IMAGEafter platform PR #590 removed an inline sidecar-image pin from the SND spec
- Dominant language
- Go
- Stars
- 1
- Forks
- 2
- Avg merge
- 2h 29m
- Merged PRs (30d)
- 56
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from sei-protocol/sei-k8s-controller
-
Difficulty 5/5 Over a week Newbie friendliness 32/100
-
Difficulty 5/5 Over a week Newbie friendliness 32/100
sei-protocol/sei-k8s-controller#457 · 2 comments ·
-
Difficulty 3/5 1-2 days Newbie friendliness 68/100
-
Difficulty 4/5 3-5 days Newbie friendliness 45/100
-
Difficulty 3/5 1-2 days Newbie friendliness 65/100
All issues in sei-protocol/sei-k8s-controller
Similar issues
-
optimization optimization:agents-md-curator
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
githubnext/gh-aw-cao#13143 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
blinklabs-io/bursa#904 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
yanet-platform/ipfw-go#129 ·
-
bug confmap/provider/googlesecretmanagerprovider needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
open-telemetry/opentelemetry-collector-contrib#51273 · 2 comments ·
-
bug: AI Gateway client filter lists "Unknown" twice when NULL and literal Unknown clients coexist Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 90/100