Celery Backend Implementation
Open
@MarcusHsieh is already working on this.
Since Sep 4, 2025.
Merlin 2.0
refactor
- Dominant language
- Python
- Stars
- 151
- Forks
- 30
- PR merge metrics
- No merged PRs in 30d
Description
Complete TaskServerInterface implementation wrapping existing Celery functionality.
As #538 discusses, there are potential race conditions with Celery's add_to_chord call when running at scale. I tried adding a locking mechanism to prevent this but it simply makes the workflows run way too slow. The next step to resolve this is to refactor the entire tasks.py file so that it no longer uses chords (or at least no longer uses add_to_chord).
The benefits from this will hopefully be:
- No more disappearing tasks
- The ability to support more backends than just Redis
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.