Planner does not re-fire `apply-statefulset` when StatefulSet is deleted post-bootstrap

Open
#284 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
58/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Quiet
Tech stack
go, kubernetes

Research direction

Start with the SeiNode reconciler and planner path described in the issue, focusing on the post-bootstrap phase: Running handling and the apply-statefulset, apply-service, and apply-rbac-proxy-config tasks. Trace how derived resources are currently checked and how chain-upgrade and peer re-discovery plans are triggered. Done means a missing StatefulSet is recreated from current SeiNode.Spec without rerunning bootstrap work or deleting the data PVC, while existing reconcile paths remain unchanged.

Written by the indexing model from the issue text.

Description

Problem

When a SeiNode reaches phase: Running, the controller's planner is finished. If the rendered StatefulSet is later deleted (manually for ops reasons, or as part of an aftercare sweep), the SeiNode reconciler does not detect the missing derived resource and re-fire the apply-statefulset task. The SeiNode stays in phase: Running indefinitely with no live pod.

Live-reproduced today against the state-size-analyzer SND in pacific-1: deleted the StatefulSet to force a sidecar-image re-render via the platform-default SEI_SIDECAR_IMAGE env. Controller never recreated it. Workaround required deleting the SeiNode itself so the SND's reconcileSeiNodes would recreate it fresh and the bootstrap plan would re-run from scratch — which also wipes the data PVC (SeiNode-owned), forcing a full state-sync redo.

Impact

Any ops procedure that deletes the StatefulSet becomes a one-way door — the SND won't bring seid back automatically. The most immediate consumer is the state-size-analysis CronJob (queued follow-up to platform state-size-analyzer.yaml) which is designed to scale replicas: 0, run an analyzer Job against the released PVC, and scale back — that pattern's correctness depends on the controller re-creating the StatefulSet on the scale-up path. Today operators have to know the "delete the SeiNode and accept PVC loss" workaround, which is a footgun: silent state loss for anyone who doesn't realize the cascade.

Relevant experts

  • kubernetes-specialist — controller planner + reconcile logic

Proposed approach

In the SeiNode reconciler, after the bootstrap plan completes (phase: Running), continue to assert derived resources exist on each reconcile pass. If the StatefulSet matching SeiNode.Name is missing, fire a new plan containing only the post-bootstrap apply tasks (apply-statefulset, apply-service, apply-rbac-proxy-config if TLS is enabled). Do not re-fire discover-peers / configure-state-sync / config-validate — those already ran and the existing PVC carries their result. The new plan should be a targeted "rebuild-derived-resources" flow, not a full bootstrap.

Acceptance criteria

  • Deleting the rendered StatefulSet on a Running SeiNode causes the controller to recreate it within one reconcile cycle
  • The new StatefulSet is rendered from current SeiNode.Spec (so any sidecar-image change since initial bootstrap is picked up via the platform default)
  • The data PVC is preserved across the recreate — no state-sync redo
  • Existing reconcile paths (chain-upgrade plan, peer re-discovery) remain unaffected

Out of scope

The related "ensure-data-pvc fails terminally when a stale orphaned PVC is present from a just-deleted SeiNode" race condition. That's a separate issue — that code path needs either auto-adoption logic or retry-with-backoff so K8s GC has time to catch up on the orphan. Filing separately if/when it comes up again.

References

  • Live reproduction today on the pacific-1/state-size-analyzer SND
  • Triggering context: cycling the pod to pick up the platform-default SEI_SIDECAR_IMAGE after platform PR #590 removed an inline sidecar-image pin from the SND spec
Dominant language
Go
Stars
1
Forks
2
Avg merge
2h 29m
Merged PRs (30d)
56

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from sei-protocol/sei-k8s-controller

All issues in sei-protocol/sei-k8s-controller

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.