plotly / plotly/dash

Add Filesystem as a backend for CeleryManager in background callbacks

Open
#3,116 0 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

cs feature P2
Dominant language
Python
Stars
24.4k
Forks
2.3k
Avg merge
2d 7h
Merged PRs (30d)
13

Description

How is this feature useful?

  • When the value that the background callback returns is bigger than 512MB (formatted as a string), an error will be raised: celery.exceptions.BackendStoreError: value too large for Redis backend
  • This error occurs because the CeleryManager temporarily stores the output value of the background callback.
  • A workaround for this could be using a FilesystemBackend (for example, the DE5 Persistent Filesystem), like this:
celery_app = Celery(__name__, broker=os.environ['REDIS_URL'], backend='../mount')
background_callback_manager = CeleryManager(celery_app)

What's the current state?

  • In the CeleryManager for background callbacks, the values are stored as strings (ref code: link).
  • When the backend is a filesystem, celery (as a dependency of the CeleryManager) expects the format to be bytes (ref: link).
  • The current version of dash CeleryManager for background callbacks doesn't contemplate the case where the backend is a filesystem; it assumes Redis will be used as a backend (ref: link).
  • Celery does contemplate the option os using a FilesystemBackend (ref link, but the CeleryManager doesn't know how to handle that situation.

What changes would be necessary?

  • In places where cache.set(progress_key, json.dumps(progress_value, cls=PlotlyJSONEncoder)) is used directly add a conditional logic to support both RedisBackend and FilesystemBackend like:
from kombu.utils.encoding import str_to_bytes [[source]](https://docs.celeryq.dev/projects/kombu/en/latest/_modules/kombu/utils/encoding.html#str_to_bytes)
...

if isinstance(celery_app.backend, RedisBackend):
    cache.set(progress_key, json.dumps(progress_value, cls=PlotlyJSONEncoder))
elif isinstance(celery_app.backend, FilesystemBackend):
    celery_app.backend.set(progress_key, str_to_bytes(json.dumps(progress_value(progress_value)))
    # ref: https://github.com/celery/celery/blob/main/celery/backends/filesystem.py#L87

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in dash/long_callback/managers/celery_manager.py, especially the direct cache.set calls described in the issue. Compare the RedisBackend and FilesystemBackend behavior in the linked Celery sources; done means CeleryManager can store background-callback values with either backend without breaking the existing Redis path.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, redis
Domain
backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.