Bogdanp / Bogdanp/dramatiq

Discussion: Resque-like queue prioritization

Aperta
#519 5 commenti 5 reazioni 0 assegnatari Vedi su GitHub
enhancement
Lingua principale
Python
Stelle
5.3k
Fork
383
Merge medio
9h 41m
PR unite (30g)
2

Descrizione

# Situation
The default behavior of Dramatiq when multiple queues are present (either all of them or a subset), is to fetch data from them using the consumer threads. This in practice means that all queues are drained equally (assuming they all have messages in them).

In contrast, [Resque](https://github.com/resque/resque), grabs messages in the order the queues are set when starting the workers. For example, given the queues `low`, `high` and `default`, one could start a worker with `--queues=high,default,low` and the queues would be drained in order. This ofc could lead to the `low` queue being starved if `high` or `default` always had a constant influx of new messages. To overcome this, often there would be multiple workers with different settings like `--queues=low,default,high` to ensure that all queues had at least one workers dedicated to them.

When compared to dramatiq, resque allows creating dedicated workers for a given queue, that when idle will try to process jobs from other queues. Dramatiq can have dedicated workers for a queue, however, they will stay idle if that queue is empty.

## Potential Solutions
### Middleware

Since dramatiq has good middleware support, one could write a new middleware to implement this behavior. However, it seems that there aren't any available hooks that could be used to handle this in an efficient way.

Ideally, no tasks should be waiting in a worker, while a potential another worker could be handling it. We should also avoid taking tasks off a queue, and then adding it back, repeatedly.

### Builtin

Changing how dramatiq works probably isn't the best approach, since it could break setups that are functional with the current behavior. We could have a setting to toggle this behavior. Something like `--drain-queues-in-order`. IMO this solution is less preferred, since it adds more complexity and a somewhat opinionated way of how things should work.

## Questions

0. Is anyone else interested in this? Is this even useful or desired?
1. Can any of the current hooks provided by the middleware architecture allow this to be implemented in a reasonable manner? Preferentially without changing dramatiq itself
2. If not, what do we need in order do implement it the "middleware way"?

I am trying to get an MVP working using the middleware approach, but so far nothing. Still getting familiar with dramatiq's internals

Note: I am starting the discussion here because the reddit seems to be abandoned

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.