dragonflydb / dragonflydb/dragonfly-operator
The Dragonfly master does not switch when a node fails in Kubernetes.
- Dominant language
- Go
- Stars
- 357
- Forks
- 111
- PR merge metrics
- No merged PRs in 30d
Description
**Describe the bug**
There is Dragonfly running in Kubernetes (1 master, 2 replicas).
The second time during issue with node, the master does not failover to the slave.
**To Reproduce**
Steps to reproduce the behavior:
1. Run dragonfly in k8s
2. Check in which node dragonfly master is running
3. Shutdown the node
**Expected behavior**
New master should be elected, if the master is unavailable.
**Environment:**
- Workers OS: Ubuntu 22.04.4 LTS
- Workers Kernel: 5.15.0-105-generic
- Containerized: Kubernetes v1.29.2
- dragonfly v1.26.2-096fde172300de91850c42dab24aa09ffee254d0
- image: docker.dragonflydb.io/dragonflydb/operator:v1.1.9
**Reproducible Code Snippet**
```
k get po -l master
shutdown now # node where pod with master is running
# wait
k get po -l master # (no master)
```
**Additional context**
Master pod in `pending` state:
```$ k get po | grep dragonfly-foo
dragonfly-foo-0 0/1 Pending 0 54m
dragonfly-foo-1 1/1 Running 0 22d
dragonfly-foo-2 1/1 Running 0 22d
```
Info from slaves:
```
root@dragonfly-foo-1:/data# redis-cli
127.0.0.1:6379> info replication
# Replication
role:slave
master_host:172.20.31.160
master_port:9999
master_link_status:down
master_last_io_seconds_ago:2809
master_sync_in_progress:0
master_replid:779aexec8fa961f6e7a431b46bb37ff4d86c41bc
slave_priority:100
slave_read_only:1
root@dragonfly-foo-2:/data# redis-cli
127.0.0.1:6379> info replication
# Replication
role:slave
master_host:172.20.31.160
master_port:9999
master_link_status:down
master_last_io_seconds_ago:2829
master_sync_in_progress:0
master_replid:779aexec8fa961f6e7a431b46bb37ff4d86c41bc
slave_priority:100
slave_read_only:1
```
And in the pod logs just:
```
W20250311 14:38:34.344835 11 replica.cc:217] Error connecting to 172.20.31.160:9999 system:125
W20250311 14:38:35.845142 11 replica.cc:217] Error connecting to 172.20.31.160:9999 system:125
```
Contributor guide
Assessment
This issue has not been assessed yet.