dragonflydb / dragonflydb/dragonfly-operator
REPLTAKEOVER client read timeout shorter than the 10000s timeout it sends
- Dominant language
- Go
- Stars
- 357
- Forks
- 111
- PR merge metrics
- No merged PRs in 30d
Description
Hit this during a routine image bump on a 3-replica Dragonfly. The rolling update wedged the whole cluster for hours instead of failing and retrying, and I think the cause is a timeout mismatch in the operator.
`replTakeover` sends `REPLTAKEOVER 10000`, so the server locks the database down for up to 10000 seconds:
https://github.com/dragonflydb/dragonfly-operator/blob/v1.6.1/internal/controller/dragonfly_instance.go#L1124
But the client running that command has `ReadTimeout: 10 * time.Second`:
https://github.com/dragonflydb/dragonfly-operator/blob/v1.6.1/internal/controller/dragonfly_instance.go#L70-L79
So if the takeover takes longer than 10 seconds, the operator gives up, and the server stays locked down (for me it was approx. ~2h45m). Nothing is left waiting to clean it up. The promoted replica still answers PING and `INFO stats` but hangs on `INFO` and `INFO replication`, the old master sits in TAKEN_OVER after logging "Takeover initiated, locking down the database.", and the data port returns `LOADING Dragonfly is loading the dataset in memory` the whole time.
I don't think this is specific to the version I was upgrading to. Anything that makes catch-up take more than 10 seconds should trigger it.
## Steps to reproduce
1. Deploy a `Dragonfly` with 3 replicas (mine: `--cluster_mode=emulated`, `--proactor_threads=2`, 512Mi limit per pod).
2. Change `spec.image` to trigger a rolling update.
3. Make the takeover on the promoted replica take longer than 10 seconds.
4. The operator logs a timeout on the REPLTAKEOVER call, but the cluster stays down. `INFO` on the promoted replica's admin port hangs, and clients on 6379 get `LOADING`.
Recovery needed manual intervention: revert the image tag and delete the stuck replica pod. Doing `REPLICAOF NO ONE` on a healthy replica and deleting the master pod also worked.
## Environment
- Operator `v1.6.1`
- Dragonfly `v1.40.1` -> `v1.40.2`
- 3 replicas, Kubernetes on bare metal
It looks like #559 was going in this direction before it was closed. `main` still has the hardcoded `10000`.
Happy to follow up with more detail, logs, or testing if that would help.
Contributor guide
Research direction
Start in internal/controller/dragonfly_instance.go at the ReadTimeout configuration around lines 70-79 and the REPLTAKEOVER call around line 1124. Reproduce the issue with a three-replica rolling image update and a takeover lasting over 10 seconds; done means a slow takeover no longer leaves the cluster locked down and unavailable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- databases, devops
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 74/100