Add Filesystem as a backend for CeleryManager in background callbacks
Open
Nobody has claimed this yet.
cs
feature
P2
- Dominant language
- Python
- Stars
- 24.4k
- Forks
- 2.3k
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 13
Description
How is this feature useful?
- When the value that the background callback returns is bigger than 512MB (formatted as a string), an error will be raised:
celery.exceptions.BackendStoreError: value too large for Redis backend - This error occurs because the CeleryManager temporarily stores the output value of the background callback.
- A workaround for this could be using a FilesystemBackend (for example, the DE5 Persistent Filesystem), like this:
celery_app = Celery(__name__, broker=os.environ['REDIS_URL'], backend='../mount')
background_callback_manager = CeleryManager(celery_app)
What's the current state?
- In the CeleryManager for background callbacks, the values are stored as strings (ref code: link).
- When the backend is a filesystem, celery (as a dependency of the CeleryManager) expects the format to be bytes (ref: link).
- The current version of dash CeleryManager for background callbacks doesn't contemplate the case where the backend is a filesystem; it assumes Redis will be used as a backend (ref: link).
- Celery does contemplate the option os using a FilesystemBackend (ref link, but the CeleryManager doesn't know how to handle that situation.
What changes would be necessary?
- In places where
cache.set(progress_key, json.dumps(progress_value, cls=PlotlyJSONEncoder))is used directly add a conditional logic to support both RedisBackend and FilesystemBackend like:
from kombu.utils.encoding import str_to_bytes [[source]](https://docs.celeryq.dev/projects/kombu/en/latest/_modules/kombu/utils/encoding.html#str_to_bytes)
...
if isinstance(celery_app.backend, RedisBackend):
cache.set(progress_key, json.dumps(progress_value, cls=PlotlyJSONEncoder))
elif isinstance(celery_app.backend, FilesystemBackend):
celery_app.backend.set(progress_key, str_to_bytes(json.dumps(progress_value(progress_value)))
# ref: https://github.com/celery/celery/blob/main/celery/backends/filesystem.py#L87
- If necessary - add some logic that detects if the value passed to the backend is a Redis database or a filepath > It doesn't seem to be necessary since Celery detects this automatically: https://github.com/celery/celery/blob/main/celery/app/base.py#L1092
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in dash/long_callback/managers/celery_manager.py, especially the direct cache.set calls described in the issue. Compare the RedisBackend and FilesystemBackend behavior in the linked Celery sources; done means CeleryManager can store background-callback values with either backend without breaking the existing Redis path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, redis
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100