antirez / antirez/disque

[Request] Queue throttling

Abierto
#143 1 comentario 2 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
C
Estrellas
8.1k
Forks
532
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

Firstly, disque is great! \o/

Secondly are there any plans to implement queue throttling or some variation of it. I'll explain:

For example:
X public web API has a limit of 100 http requests per 10 secs, and I use a single job to consume one of these requests.

Now normally in Redis I would achieve this using a mix of `EXPIRE` and `INCR` - so I set the ttl to 10 seconds and INCR on each successful job run.

Here's a rough redis lua implementation of the above, to demonstrate:

``` lua
--[[
key 1 -> key name - ip, user id or some unique key to throttle X by
arg 1 -> limit
arg 2 -> seconds

returns {
throttled: -> 1 if should throttle, 0 if still within limit
remaining: -> how many reqs left until throttled,
ttl: -> seconds remaining until limit resets
}
]]

local count = redis.call('INCR', KEYS[1])
local ttl = redis.call('ttl',KEYS[1])

if count == 1 or ttl == -1 then
redis.call('EXPIRE', KEYS[1], ARGV[2])
end

if ttl == -1 then
ttl = tonumber(ARGV[2])
end

if count > tonumber(ARGV[1]) then
return {1,0,ttl}
end

return {0,tonumber(ARGV[1]) - count,ttl}
```

On a single queue job consumer this wouldn't be a problem really, could just keep counts locally, but I currently run a farm of queue workers (using redis), so would be no easy way to maintain the counts across all the servers without using redis alongside disque.

On `ADDJOB` I see there is `DELAY` and `MAXLEN` which is a good start though :+1:

Possible solutions:
1. Allow storing arbitrary data onto queues, then let the clients handle limits / any other use cases for storing queue specific data.
2. Allow storing arbitrary data on disque, same as above but not queue specific - probably better due to queues getting deleted when empty. Doesn't need to be another redis, just the basic get / set sort of stuff?
3. Native feature of disque?

Or any other suggestions? Not sure if I'm the only one with this use case though? :hear_no_evil:

Thanks,
Mike

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.