dragonflydb / dragonflydb/dragonfly

DragonflyDb not working with python dramatiq workers

Open
#3,962 5 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
C++
Stars
31.5k
Forks
1.3k
Avg merge
1d 10h
Merged PRs (30d)
137

Description

**Describe the bug**
I am running dragonflydb as our preferred cache in place of redis, everything works as expected until we introduced `dramatiq` task workers as we'd wanted to relief the longer tasks to a seperate process. The task workers started failing with this error
`redis.exceptions.ResponseError: Error running script (call to 976fb89fbb3c404c8c7bed59060c56521eb9ba69): @user_script:59: script tried accessing undeclared key, key: dramatiq:__heartbeats__`

**To Reproduce**
Steps to reproduce the behavior:
1. Run the `docker.dragonflydb.io/dragonflydb/dragonfly` docker image
2. Install and spin up dramatiq task workers `dramatiq -p 4 -t 4 actors` connected to the dragonfly instance

**Expected behavior**
I expect the worker to start up with it specified configurations, N/B this works on Redis

Stack trace:
`ado-general-worker | [2024-10-22 13:05:37,328] [PID 27] [Thread-4] [dramatiq.worker.ConsumerThread(general.DQ)] [INFO] Restarting consumer in 3.00 seconds.
trado-general-worker | [2024-10-22 13:05:37,328] [PID 26] [Thread-3] [dramatiq.worker.ConsumerThread(general)] [CRITICAL] Consumer encountered an unexpected error.
trado-general-worker | Traceback (most recent call last):
trado-general-worker | File "/usr/local/lib/python3.12/site-packages/dramatiq/brokers/redis.py", line 344, in __next__
trado-general-worker | data = self.message_cache.pop(0)
trado-general-worker | ^^^^^^^^^^^^^^^^^^^^^^^^^
trado-general-worker | IndexError: pop from empty list
trado-general-worker |
trado-general-worker | During handling of the above exception, another exception occurred:
trado-general-worker |
trado-general-worker | Traceback (most recent call last):
trado-general-worker | File "/usr/local/lib/python3.12/site-packages/dramatiq/worker.py", line 265, in run
trado-general-worker | for message in self.consumer:
trado-general-worker | ^^^^^^^^^^^^^
trado-general-worker | File "/usr/local/lib/python3.12/site-packages/dramatiq/brokers/redis.py", line 363, in __next__
trado-general-worker | self.message_cache = messages = self.broker.do_fetch(
trado-general-worker | ^^^^^^^^^^^^^^^^^^^^^
trado-general-worker | File "/usr/local/lib/python3.12/site-packages/dramatiq/brokers/redis.py", line 281, in do_dispatch
trado-general-worker | return dispatch(args=args, keys=keys)
trado-general-worker | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
trado-general-worker | File "/usr/local/lib/python3.12/site-packages/redis/commands/core.py", line 5931, in __call__
trado-general-worker | return client.evalsha(self.sha, len(keys), *args)
trado-general-worker | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
trado-general-worker | File "/usr/local/lib/python3.12/site-packages/redis/commands/core.py", line 5307, in evalsha
trado-general-worker | return self._evalsha("EVALSHA", sha, numkeys, *keys_and_args)
trado-general-worker | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
trado-general-worker | File "/usr/local/lib/python3.12/site-packages/redis/commands/core.py", line 5291, in _evalsha
trado-general-worker | return self.execute_command(command, sha, numkeys, *keys_and_args)
trado-general-worker | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
trado-general-worker | File "/usr/local/lib/python3.12/site-packages/redis/client.py", line 508, in execute_command
trado-general-worker | return conn.retry.call_with_retry(
trado-general-worker | ^^^^^^^^^^^^^^^^^^^^^^^^^^^
trado-general-worker | File "/usr/local/lib/python3.12/site-packages/redis/retry.py", line 46, in call_with_retry
trado-general-worker | return do()
trado-general-worker | ^^^^
trado-general-worker | File "/usr/local/lib/python3.12/site-packages/redis/client.py", line 509, in
trado-general-worker | lambda: self._send_command_parse_response(
trado-general-worker | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
trado-general-worker | File "/usr/local/lib/python3.12/site-packages/redis/client.py", line 485, in _send_command_parse_response
trado-general-worker | return self.parse_response(conn, command_name, **options)
trado-general-worker | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
trado-general-worker | File "/usr/local/lib/python3.12/site-packages/redis/client.py", line 525, in parse_response
trado-general-worker | response = connection.read_response()
trado-general-worker | ^^^^^^^^^^^^^^^^^^^^^^^^^^
trado-general-worker | File "/usr/local/lib/python3.12/site-packages/redis/connection.py", line 516, in read_response
trado-general-worker | raise response
trado-general-worker | redis.exceptions.ResponseError: Error running script (call to 976fb89fbb3c404c8c7bed59060c56521eb9ba69): @user_script:59: script tried accessing undeclared key, key: dramatiq:__heartbeats__`

**Environment (please complete the following information):**
- OS: [ubuntu 20.04]
- Containerized?: [Docker, Docker Compose]
- Dragonfly Version: [latest]

**Reproducible Code Snippet**
```
@dramatiq.actor(queue_name="usermgmt")
async def foo(
args ...
):
```

Contributor guide

Open the contributing guide

Research direction

Start with the Docker Dragonfly image and the dramatiq Redis broker path shown in the stack trace, especially the dispatch script that accesses dramatiq:__heartbeats__. Reproduce the failure with four processes and four threads on Ubuntu 20.04, then verify that the worker starts with the same configuration and no undeclared-key error.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python, redis
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.