dragonflydb / dragonflydb/dragonfly-operator

The Dragonfly master does not switch when a node fails in Kubernetes.

Open
#302 17 comments 3 reactions 1 assignee Claimed by @Pothulapati View on GitHub
bug
Dominant language
Go
Stars
357
Forks
111
PR merge metrics
No merged PRs in 30d

Description

**Describe the bug**
There is Dragonfly running in Kubernetes (1 master, 2 replicas).
The second time during issue with node, the master does not failover to the slave.

**To Reproduce**
Steps to reproduce the behavior:
1. Run dragonfly in k8s
2. Check in which node dragonfly master is running
3. Shutdown the node

**Expected behavior**
New master should be elected, if the master is unavailable.

**Environment:**
- Workers OS: Ubuntu 22.04.4 LTS
- Workers Kernel: 5.15.0-105-generic
- Containerized: Kubernetes v1.29.2
- dragonfly v1.26.2-096fde172300de91850c42dab24aa09ffee254d0
- image: docker.dragonflydb.io/dragonflydb/operator:v1.1.9

**Reproducible Code Snippet**
```
k get po -l master
shutdown now # node where pod with master is running
# wait
k get po -l master # (no master)
```

**Additional context**
Master pod in `pending` state:
```$ k get po | grep dragonfly-foo
dragonfly-foo-0 0/1 Pending 0 54m
dragonfly-foo-1 1/1 Running 0 22d
dragonfly-foo-2 1/1 Running 0 22d
```
Info from slaves:
```
root@dragonfly-foo-1:/data# redis-cli
127.0.0.1:6379> info replication
# Replication
role:slave
master_host:172.20.31.160
master_port:9999
master_link_status:down
master_last_io_seconds_ago:2809
master_sync_in_progress:0
master_replid:779aexec8fa961f6e7a431b46bb37ff4d86c41bc
slave_priority:100
slave_read_only:1

root@dragonfly-foo-2:/data# redis-cli
127.0.0.1:6379> info replication
# Replication
role:slave
master_host:172.20.31.160
master_port:9999
master_link_status:down
master_last_io_seconds_ago:2829
master_sync_in_progress:0
master_replid:779aexec8fa961f6e7a431b46bb37ff4d86c41bc
slave_priority:100
slave_read_only:1
```
And in the pod logs just:
```
W20250311 14:38:34.344835 11 replica.cc:217] Error connecting to 172.20.31.160:9999 system:125
W20250311 14:38:35.845142 11 replica.cc:217] Error connecting to 172.20.31.160:9999 system:125
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.