dask / dask/distributed

distributed Dask queue : producer consumer

Open
#5,843 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Hello, 😄

I'm a data scientist/data engineer working on a specific workflow where I need to process a huge amount of documents so i tried using `distributed.Queue` to right a producer/consumer pattern using Dask distributed.

I might miss something about how to correctly implement the producer/consumer using distributed but here are some important features that could be added to `distributed.Queue`. I would also like to help writing this class, so thanks for your guidance :

- [ ] The queue is not garbage collected, calling `del q` doesnt actually free up the distributed memory
- [ ] We might need to implement a `q.join()` in distributed manner to block until all the queue items have been processed
- [ ] Batching in queue doesnt have a '_last object mode'_ where we return lesser objects than the `batch_size` if the queue is empty
- [ ] Spawning a "pool of consumers" in a distributed system can be tricky : basically the machines producing are also the ones waiting for the tasks to be added ? Is there a way to load balance producers and consumers in the dask scheduler ?

Thanks a lot and I hope these features make sens. I hope, I can be of some help in advancing this class !

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.