antirez / antirez/disque

[Request] Queue throttling

Aperta
#143 1 commento 2 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
C
Stelle
8.1k
Fork
532
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

Firstly, disque is great! \o/

Secondly are there any plans to implement queue throttling or some variation of it. I'll explain:

For example:
X public web API has a limit of 100 http requests per 10 secs, and I use a single job to consume one of these requests.

Now normally in Redis I would achieve this using a mix of `EXPIRE` and `INCR` - so I set the ttl to 10 seconds and INCR on each successful job run.

Here's a rough redis lua implementation of the above, to demonstrate:

``` lua
--[[
key 1 -> key name - ip, user id or some unique key to throttle X by
arg 1 -> limit
arg 2 -> seconds

returns {
throttled: -> 1 if should throttle, 0 if still within limit
remaining: -> how many reqs left until throttled,
ttl: -> seconds remaining until limit resets
}
]]

local count = redis.call('INCR', KEYS[1])
local ttl = redis.call('ttl',KEYS[1])

if count == 1 or ttl == -1 then
redis.call('EXPIRE', KEYS[1], ARGV[2])
end

if ttl == -1 then
ttl = tonumber(ARGV[2])
end

if count > tonumber(ARGV[1]) then
return {1,0,ttl}
end

return {0,tonumber(ARGV[1]) - count,ttl}
```

On a single queue job consumer this wouldn't be a problem really, could just keep counts locally, but I currently run a farm of queue workers (using redis), so would be no easy way to maintain the counts across all the servers without using redis alongside disque.

On `ADDJOB` I see there is `DELAY` and `MAXLEN` which is a good start though :+1:

Possible solutions:
1. Allow storing arbitrary data onto queues, then let the clients handle limits / any other use cases for storing queue specific data.
2. Allow storing arbitrary data on disque, same as above but not queue specific - probably better due to queues getting deleted when empty. Doesn't need to be another redis, just the basic get / set sort of stuff?
3. Native feature of disque?

Or any other suggestions? Not sure if I'm the only one with this use case though? :hear_no_evil:

Thanks,
Mike

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.