dragonflydb / dragonflydb/dragonfly-operator

REPLTAKEOVER client read timeout shorter than the 10000s timeout it sends

Open
#582 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
357
Forks
111
PR merge metrics
No merged PRs in 30d

Description

Hit this during a routine image bump on a 3-replica Dragonfly. The rolling update wedged the whole cluster for hours instead of failing and retrying, and I think the cause is a timeout mismatch in the operator.

`replTakeover` sends `REPLTAKEOVER 10000`, so the server locks the database down for up to 10000 seconds:

https://github.com/dragonflydb/dragonfly-operator/blob/v1.6.1/internal/controller/dragonfly_instance.go#L1124

But the client running that command has `ReadTimeout: 10 * time.Second`:

https://github.com/dragonflydb/dragonfly-operator/blob/v1.6.1/internal/controller/dragonfly_instance.go#L70-L79

So if the takeover takes longer than 10 seconds, the operator gives up, and the server stays locked down (for me it was approx. ~2h45m). Nothing is left waiting to clean it up. The promoted replica still answers PING and `INFO stats` but hangs on `INFO` and `INFO replication`, the old master sits in TAKEN_OVER after logging "Takeover initiated, locking down the database.", and the data port returns `LOADING Dragonfly is loading the dataset in memory` the whole time.

I don't think this is specific to the version I was upgrading to. Anything that makes catch-up take more than 10 seconds should trigger it.

## Steps to reproduce

1. Deploy a `Dragonfly` with 3 replicas (mine: `--cluster_mode=emulated`, `--proactor_threads=2`, 512Mi limit per pod).
2. Change `spec.image` to trigger a rolling update.
3. Make the takeover on the promoted replica take longer than 10 seconds.
4. The operator logs a timeout on the REPLTAKEOVER call, but the cluster stays down. `INFO` on the promoted replica's admin port hangs, and clients on 6379 get `LOADING`.

Recovery needed manual intervention: revert the image tag and delete the stuck replica pod. Doing `REPLICAOF NO ONE` on a healthy replica and deleting the master pod also worked.

## Environment

- Operator `v1.6.1`
- Dragonfly `v1.40.1` -> `v1.40.2`
- 3 replicas, Kubernetes on bare metal

It looks like #559 was going in this direction before it was closed. `main` still has the hardcoded `10000`.

Happy to follow up with more detail, logs, or testing if that would help.

Contributor guide

Open the contributing guide

Research direction

Start in internal/controller/dragonfly_instance.go at the ReadTimeout configuration around lines 70-79 and the REPLTAKEOVER call around line 1124. Reproduce the issue with a three-replica rolling image update and a takeover lasting over 10 seconds; done means a slow takeover no longer leaves the cluster locked down and unavailable.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
databases, devops
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.