dragonflydb / dragonflydb/dragonfly-operator
Operator may leave pods without role label and hit metadata patch conflicts during load/rollout
- Dominant language
- Go
- Stars
- 357
- Forks
- 111
- PR merge metrics
- No merged PRs in 30d
Description
### Problem
In some reconciliation paths, pod role metadata in Kubernetes can drift from actual Dragonfly replication state:
- Pod is already replicating correctly, but role label is missing/stale.
- Reconcile may stall on `!podReady` while pod is still loading data.
- Pod metadata updates can fail with optimistic-concurrency conflicts: `Operation cannot be fulfilled ... object has been modified`.
This can leave pods unlabeled or temporarily misclassified and increases reconciliation instability during rollouts/failovers.
### Expected behavior
Pod role labels converge to actual Redis replication state quickly and idempotently.
Topology correction can run as soon as admin socket is reachable (not blocked on full data load).
Metadata patching is robust under concurrent status updates; no persistent conflict errors.
### Proposed fix
Implemented in [PR #529](https://github.com/dragonflydb/dragonfly-operator/pull/529):
- Add conflict-resilient pod metadata patch helper (RetryOnConflict + re-GET + merge patch).
- Add idempotent role reconciliation path based on Redis INFO replication (label-only patch when already correct).
- Use reachability gate (isReachable) for topology mutation paths.
- Reorder lifecycle reconciliation so missing role-label reconciliation runs before !podReady short-circuit.
- Requeue transient replication-cancelled cases instead of surfacing as terminal reconcile failures.
@miledxz, could you please take a look, thanks a lot
Contributor guide
Assessment
This issue has not been assessed yet.