antirez / antirez/disque

[Request] Queue throttling

オープン
#143 コメント 1 件 リアクション 2 件 担当者 0 名 GitHub で見る
主要言語
C
スター
8.1k
フォーク
532
PR マージ指標
30日以内にマージされた PR はありません

説明

Firstly, disque is great! \o/

Secondly are there any plans to implement queue throttling or some variation of it. I'll explain:

For example:
X public web API has a limit of 100 http requests per 10 secs, and I use a single job to consume one of these requests.

Now normally in Redis I would achieve this using a mix of `EXPIRE` and `INCR` - so I set the ttl to 10 seconds and INCR on each successful job run.

Here's a rough redis lua implementation of the above, to demonstrate:

``` lua
--[[
key 1 -> key name - ip, user id or some unique key to throttle X by
arg 1 -> limit
arg 2 -> seconds

returns {
throttled: -> 1 if should throttle, 0 if still within limit
remaining: -> how many reqs left until throttled,
ttl: -> seconds remaining until limit resets
}
]]

local count = redis.call('INCR', KEYS[1])
local ttl = redis.call('ttl',KEYS[1])

if count == 1 or ttl == -1 then
redis.call('EXPIRE', KEYS[1], ARGV[2])
end

if ttl == -1 then
ttl = tonumber(ARGV[2])
end

if count > tonumber(ARGV[1]) then
return {1,0,ttl}
end

return {0,tonumber(ARGV[1]) - count,ttl}
```

On a single queue job consumer this wouldn't be a problem really, could just keep counts locally, but I currently run a farm of queue workers (using redis), so would be no easy way to maintain the counts across all the servers without using redis alongside disque.

On `ADDJOB` I see there is `DELAY` and `MAXLEN` which is a good start though :+1:

Possible solutions:
1. Allow storing arbitrary data onto queues, then let the clients handle limits / any other use cases for storing queue specific data.
2. Allow storing arbitrary data on disque, same as above but not queue specific - probably better due to queues getting deleted when empty. Doesn't need to be another redis, just the basic get / set sort of stuff?
3. Native feature of disque?

Or any other suggestions? Not sure if I'm the only one with this use case though? :hear_no_evil:

Thanks,
Mike

コントリビューションガイド

このリポジトリのコントリビューションガイドは索引されていません

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。