feat(planner): detect sidecar.image drift on Running nodes

Open
#167 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
52/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Quiet
Tech stack
go, kubernetes

Research direction

Start with internal/planner/planner.go:651-658 and compare buildNodeUpdatePlan at :686-694, then read internal/planner/node_update_test.go and internal/planner/doc.go:37. Trace how the latest pod or StatefulSet image is observed before choosing the sidecar drift path. Done means a Running SeiNode sidecar-only image change builds and completes a plan, status reflects the running image, the analogous test passes, and the doc comment covers sidecar drift.

Written by the indexing model from the issue text.

Description

Problem

buildRunningPlan in internal/planner/planner.go:651-658 detects two kinds of drift on a Running SeiNode and builds nothing else:

func buildRunningPlan(node *seiv1alpha1.SeiNode) (*seiv1alpha1.TaskPlan, error) {
    if node.Spec.Image != node.Status.CurrentImage {
        return buildNodeUpdatePlan(node)            // main seid image drift
    }
    if sidecarNeedsReapproval(node) {
        return buildMarkReadyPlan(node)             // sidecar Ready=False condition
    }
    return nil, nil
}

There is no third check that compares spec.template.spec.sidecar.image against the live StatefulSet's sidecar container image. So if an operator updates only the sidecar image:

  • The CRD spec updates.
  • buildRunningPlan sees no drift (main image unchanged, sidecar Ready condition unchanged).
  • Returns nil — no plan, no rollout.
  • StatefulSet on disk still has the old sidecar image.
  • Pods keep running the old sidecar until something else triggers a restart (operator's kubectl delete pod, a main-image bump that incidentally re-applies, or a full SND disable+re-enable).

Concrete impact: rolling out a sidecar bug fix (e.g., the seictl#123 snapshot-restore regex fix shipped in image 271ab686…) currently requires either piggybacking on a main image bump or a heavy SND re-creation. Operators wanting to upgrade only the sidecar today have no clean path.

Repro

  1. Create or take an existing Running SeiNode with spec.template.spec.sidecar.image: imageA.
  2. Edit the spec to imageB (different sidecar image, no main image change).
  3. Reconcile: no plan is built. The pod continues running imageA indefinitely.

Two design options

A. SidecarUpdate plan as a third drift check

Mirror the NodeUpdate shape:

func buildRunningPlan(node *seiv1alpha1.SeiNode) (*seiv1alpha1.TaskPlan, error) {
    if node.Spec.Image != node.Status.CurrentImage {
        return buildNodeUpdatePlan(node)
    }
    if sidecarImageDrift(node) {                // NEW
        return buildSidecarUpdatePlan(node)
    }
    if sidecarNeedsReapproval(node) {
        return buildMarkReadyPlan(node)
    }
    return nil, nil
}

Where sidecarImageDrift compares spec.template.spec.sidecar.image (or default if unset) to the observed sidecar image on the most recent pod / StatefulSet. Tracking the observed image probably wants a new status.currentSidecarImage field for symmetry with currentImage.

buildSidecarUpdatePlan task sequence: apply-statefulset → apply-service → observe-sidecar-image → mark-ready. (The observe-sidecar-image task is a small new task that polls the StatefulSet rollout and stamps currentSidecarImage.)

Pros: explicit, mirrors existing pattern, easy to reason about.

Cons: a second drift-detection field to maintain, a new task type.

B. Always reconcile the StatefulSet on every Running reconcile

Replace plan-driven apply-statefulset for steady-state with an idempotent server-side-apply on every reconcile. The kubelet handles pod rotation when the sidecar image actually changes (StatefulSet rollout policy). The drift detection becomes implicit: SSA is a no-op when nothing changed.

Pros: simpler conceptually, removes an entire class of "did we forget a drift check" bugs (any field — not just sidecar.image — would propagate).

Cons: changes the contract that "the executor only mutates owned resources during plan tasks." Less obvious where rotates come from when reading logs. Need to confirm SSA idempotency interactions with the per-task field manager (seinode-controller).

Recommendation

Option A first — smaller scope, closer to the existing pattern, easier to write a focused test for. Option B is a refactor worth its own design pass.

Acceptance criteria

  • spec.template.spec.sidecar.image change on a Running SeiNode triggers a plan that ends with the pod actually running the new sidecar image, without needing a main-image bump or SND re-creation.
  • status.currentSidecarImage (or equivalent) reflects the running sidecar.
  • Test analogous to internal/planner/node_update_test.go covers the new path.
  • Doc comment in internal/planner/doc.go mentions sidecar drift alongside main-image drift.

References

  • internal/planner/planner.go:651-658 (steady-state drift logic)
  • internal/planner/planner.go:686-694 (existing buildNodeUpdatePlan to mirror)
  • internal/planner/doc.go:37 (drift detection docstring)
  • internal/planner/node_update_test.go (test pattern to mirror)
  • Surfaced during platform fork-test work where seictl#123 needed to roll out as a sidecar-only update.
Dominant language
Go
Stars
1
Forks
2
Avg merge
2h 29m
Merged PRs (30d)
56

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from sei-protocol/sei-k8s-controller

All issues in sei-protocol/sei-k8s-controller

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.