dragonflydb / dragonflydb/dragonfly
BullMQ / Cached IP from previous Dragonfly leader
- Dominant language
- C++
- Stars
- 31.6k
- Forks
- 1.3k
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 137
Description
**Describe the bug**
Not sure if this issue is with Dragonfly, Dragonfly Operator or [BullMQ](https://www.dragonflydb.io/docs/integrations/bullmq), I use [Immich](https://github.com/immich-app/immich) which has Dragonfly as a backend deployed by DragonFly Operator.
**To Reproduce**
I have no issues until there is a leader change in Dragonfly (upgrade, node restart, etc) and Immich seems to be using a cached IP of the previous leader. The Dragonfly logs are full of:
```
W20250421 11:44:22.123201 12 main_service.cc:1374] EVALSHA b4cd4bbdf096cd8d06246080315ae81c56e05a46 10 immich_bull:library:wait immich_bull:library:active immich_bull:library:prioritized immich_bull:library:events immich_bull:library:stalled immich_bull:library:limiter immich_bull:library:delayed immich_bull:library:paused immich_bull:library:meta immich_bull:library:pc immich_bull:library: 1745235862122 \xde\x00\x03\xa5token\xd9*b204d227-a63d-4442-8c3a-d5b2621590b3:35140\xaclockDuration\xcdu0\xa7limiter\xc0 failed with reason: Error running script (call to b4cd4bbdf096cd8d06246080315ae81c56e05a46): @user_script:223: -READONLY You can't write against a read only replica.
W20250421 11:44:36.187906 11 script_mgr.cc:328] Error running script (call to b4cd4bbdf096cd8d06246080315ae81c56e05a46): @user_script:223: -READONLY You can't write against a read only replica.
W20250421 11:44:36.188086 11 main_service.cc:1374] EVALSHA b4cd4bbdf096cd8d06246080315ae81c56e05a46 10 immich_bull:search:wait immich_bull:search:active immich_bull:search:prioritized immich_bull:search:events immich_bull:search:stalled immich_bull:search:limiter immich_bull:search:delayed immich_bull:search:paused immich_bull:search:meta immich_bull:search:pc immich_bull:search: 1745235876187 \xde\x00\x03\xa5token\xd9*538e32b7-302a-4663-8ed0-6fa493e04e40:35137\xaclockDuration\xcdu0\xa7limiter\xc0 failed with reason: Error running script (call to b4cd4bbdf096cd8d06246080315ae81c56e05a46): @user_script:223: -READONLY You can't write against a read only replica.
W20250421 11:44:37.226603 12 script_mgr.cc:328] Error running script (call to b4cd4bbdf096cd8d06246080315ae81c56e05a46): @user_script:223: -READONLY You can't write against a read only replica.
W20250421 11:44:37.226760 12 main_service.cc:1374] EVALSHA b4cd4bbdf096cd8d06246080315ae81c56e05a46 10 immich_bull:library:wait immich_bull:library:active immich_bull:library:prioritized immich_bull:library:events immich_bull:library:stalled immich_bull:library:limiter immich_bull:library:delayed immich_bull:library:paused immich_bull:library:meta immich_bull:library:pc immich_bull:library: 1745235877225 \xde\x00\x03\xa5token\xd9*b204d227-a63d-4442-8c3a-d5b2621590b3:35140\xaclockDuration\xcdu0\xa7limiter\xc0 failed with reason: Error running script (call to b4cd4bbdf096cd8d06246080315ae81c56e05a46): @user_script:223: -READONLY You can't write against a read only replica.
W20250421 11:44:40.165977 11 script_mgr.cc:328] Error running script (call to b8c8950186e870a4dfcc57e460f7637cd1c25077): @user_script:195: -READONLY You can't write against a read only replica.
W20250421 11:44:40.166092 11 main_service.cc:1374] EVALSHA b8c8950186e870a4dfcc57e460f7637cd1c25077 8 immich_bull:search:stalled immich_bull:search:wait immich_bull:search:active immich_bull:search:failed immich_bull:search:stalled-check immich_bull:search:meta immich_bull:search:paused immich_bull:search:events 1 immich_bull:search: 1745235880165 30000 failed with reason: Error running script (call to b8c8950186e870a4dfcc57e460f7637cd1c25077): @user_script:195: -READONLY You can't write against a read only replica.
W20250421 11:44:40.517180 11 script_mgr.cc:328] Error running script (call to b8c8950186e870a4dfcc57e460f7637cd1c25077): @user_script:195: -READONLY You can't write against a read only replica.
```
I reached out to Immich project and they suspect it is an issue with BullMQ library. I'm an end-user, not a developer, just reporting what I see.
**Expected behavior**
Dragonfly / BullMQ library should be aware of new IP address after leader change.
**Environment (please complete the following information):**
- OS: Ubuntu 24.04.02
- Kernel: 6.8.0-57-generic
- Containerized?: Kubernetes
- Dragonfly Version: 1.29.0
**Reproducible Code Snippet**
```
# Minimal code snippet to reproduce this bug
```
**Additional context**
Add any other context about the problem here.
Contributor guide
Research direction
No project file or test is identified. Start by isolating whether the stale leader IP comes from Dragonfly, Dragonfly Operator, or BullMQ during a Kubernetes leader change; done means requests follow the new leader without EVALSHA -READONLY errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100