dragonflydb / dragonflydb/dragonfly
DragonflyDb not working with python dramatiq workers
- Dominant language
- C++
- Stars
- 31.5k
- Forks
- 1.3k
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 137
Description
**Describe the bug**
I am running dragonflydb as our preferred cache in place of redis, everything works as expected until we introduced `dramatiq` task workers as we'd wanted to relief the longer tasks to a seperate process. The task workers started failing with this error
`redis.exceptions.ResponseError: Error running script (call to 976fb89fbb3c404c8c7bed59060c56521eb9ba69): @user_script:59: script tried accessing undeclared key, key: dramatiq:__heartbeats__`
**To Reproduce**
Steps to reproduce the behavior:
1. Run the `docker.dragonflydb.io/dragonflydb/dragonfly` docker image
2. Install and spin up dramatiq task workers `dramatiq -p 4 -t 4 actors` connected to the dragonfly instance
**Expected behavior**
I expect the worker to start up with it specified configurations, N/B this works on Redis
Stack trace:
`ado-general-worker | [2024-10-22 13:05:37,328] [PID 27] [Thread-4] [dramatiq.worker.ConsumerThread(general.DQ)] [INFO] Restarting consumer in 3.00 seconds.
trado-general-worker | [2024-10-22 13:05:37,328] [PID 26] [Thread-3] [dramatiq.worker.ConsumerThread(general)] [CRITICAL] Consumer encountered an unexpected error.
trado-general-worker | Traceback (most recent call last):
trado-general-worker | File "/usr/local/lib/python3.12/site-packages/dramatiq/brokers/redis.py", line 344, in __next__
trado-general-worker | data = self.message_cache.pop(0)
trado-general-worker | ^^^^^^^^^^^^^^^^^^^^^^^^^
trado-general-worker | IndexError: pop from empty list
trado-general-worker |
trado-general-worker | During handling of the above exception, another exception occurred:
trado-general-worker |
trado-general-worker | Traceback (most recent call last):
trado-general-worker | File "/usr/local/lib/python3.12/site-packages/dramatiq/worker.py", line 265, in run
trado-general-worker | for message in self.consumer:
trado-general-worker | ^^^^^^^^^^^^^
trado-general-worker | File "/usr/local/lib/python3.12/site-packages/dramatiq/brokers/redis.py", line 363, in __next__
trado-general-worker | self.message_cache = messages = self.broker.do_fetch(
trado-general-worker | ^^^^^^^^^^^^^^^^^^^^^
trado-general-worker | File "/usr/local/lib/python3.12/site-packages/dramatiq/brokers/redis.py", line 281, in do_dispatch
trado-general-worker | return dispatch(args=args, keys=keys)
trado-general-worker | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
trado-general-worker | File "/usr/local/lib/python3.12/site-packages/redis/commands/core.py", line 5931, in __call__
trado-general-worker | return client.evalsha(self.sha, len(keys), *args)
trado-general-worker | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
trado-general-worker | File "/usr/local/lib/python3.12/site-packages/redis/commands/core.py", line 5307, in evalsha
trado-general-worker | return self._evalsha("EVALSHA", sha, numkeys, *keys_and_args)
trado-general-worker | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
trado-general-worker | File "/usr/local/lib/python3.12/site-packages/redis/commands/core.py", line 5291, in _evalsha
trado-general-worker | return self.execute_command(command, sha, numkeys, *keys_and_args)
trado-general-worker | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
trado-general-worker | File "/usr/local/lib/python3.12/site-packages/redis/client.py", line 508, in execute_command
trado-general-worker | return conn.retry.call_with_retry(
trado-general-worker | ^^^^^^^^^^^^^^^^^^^^^^^^^^^
trado-general-worker | File "/usr/local/lib/python3.12/site-packages/redis/retry.py", line 46, in call_with_retry
trado-general-worker | return do()
trado-general-worker | ^^^^
trado-general-worker | File "/usr/local/lib/python3.12/site-packages/redis/client.py", line 509, in
trado-general-worker | lambda: self._send_command_parse_response(
trado-general-worker | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
trado-general-worker | File "/usr/local/lib/python3.12/site-packages/redis/client.py", line 485, in _send_command_parse_response
trado-general-worker | return self.parse_response(conn, command_name, **options)
trado-general-worker | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
trado-general-worker | File "/usr/local/lib/python3.12/site-packages/redis/client.py", line 525, in parse_response
trado-general-worker | response = connection.read_response()
trado-general-worker | ^^^^^^^^^^^^^^^^^^^^^^^^^^
trado-general-worker | File "/usr/local/lib/python3.12/site-packages/redis/connection.py", line 516, in read_response
trado-general-worker | raise response
trado-general-worker | redis.exceptions.ResponseError: Error running script (call to 976fb89fbb3c404c8c7bed59060c56521eb9ba69): @user_script:59: script tried accessing undeclared key, key: dramatiq:__heartbeats__`
**Environment (please complete the following information):**
- OS: [ubuntu 20.04]
- Containerized?: [Docker, Docker Compose]
- Dragonfly Version: [latest]
**Reproducible Code Snippet**
```
@dramatiq.actor(queue_name="usermgmt")
async def foo(
args ...
):
```
Contributor guide
Research direction
Start with the Docker Dragonfly image and the dramatiq Redis broker path shown in the stack trace, especially the dispatch script that accesses dramatiq:__heartbeats__. Reproduce the failure with four processes and four threads on Ubuntu 20.04, then verify that the worker starts with the same configuration and no undeclared-key error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, python, redis
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100