dragonflydb / dragonflydb/dragonfly-operator

Operator may leave pods without role label and hit metadata patch conflicts during load/rollout

Open
#537 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
357
Forks
111
PR merge metrics
No merged PRs in 30d

Description

### Problem
In some reconciliation paths, pod role metadata in Kubernetes can drift from actual Dragonfly replication state:
- Pod is already replicating correctly, but role label is missing/stale.
- Reconcile may stall on `!podReady` while pod is still loading data.
- Pod metadata updates can fail with optimistic-concurrency conflicts: `Operation cannot be fulfilled ... object has been modified`.
This can leave pods unlabeled or temporarily misclassified and increases reconciliation instability during rollouts/failovers.

### Expected behavior
Pod role labels converge to actual Redis replication state quickly and idempotently.
Topology correction can run as soon as admin socket is reachable (not blocked on full data load).
Metadata patching is robust under concurrent status updates; no persistent conflict errors.

### Proposed fix
Implemented in [PR #529](https://github.com/dragonflydb/dragonfly-operator/pull/529):
- Add conflict-resilient pod metadata patch helper (RetryOnConflict + re-GET + merge patch).
- Add idempotent role reconciliation path based on Redis INFO replication (label-only patch when already correct).
- Use reachability gate (isReachable) for topology mutation paths.
- Reorder lifecycle reconciliation so missing role-label reconciliation runs before !podReady short-circuit.
- Requeue transient replication-cancelled cases instead of surfacing as terminal reconcile failures.

@miledxz, could you please take a look, thanks a lot

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.