[Request] Queue throttling
- 主要言語
- C
- スター
- 8.1k
- フォーク
- 532
- PR マージ指標
- 30日以内にマージされた PR はありません
説明
Firstly, disque is great! \o/
Secondly are there any plans to implement queue throttling or some variation of it. I'll explain:
For example:
X public web API has a limit of 100 http requests per 10 secs, and I use a single job to consume one of these requests.
Now normally in Redis I would achieve this using a mix of `EXPIRE` and `INCR` - so I set the ttl to 10 seconds and INCR on each successful job run.
Here's a rough redis lua implementation of the above, to demonstrate:
``` lua
--[[
key 1 -> key name - ip, user id or some unique key to throttle X by
arg 1 -> limit
arg 2 -> seconds
returns {
throttled: -> 1 if should throttle, 0 if still within limit
remaining: -> how many reqs left until throttled,
ttl: -> seconds remaining until limit resets
}
]]
local count = redis.call('INCR', KEYS[1])
local ttl = redis.call('ttl',KEYS[1])
if count == 1 or ttl == -1 then
redis.call('EXPIRE', KEYS[1], ARGV[2])
end
if ttl == -1 then
ttl = tonumber(ARGV[2])
end
if count > tonumber(ARGV[1]) then
return {1,0,ttl}
end
return {0,tonumber(ARGV[1]) - count,ttl}
```
On a single queue job consumer this wouldn't be a problem really, could just keep counts locally, but I currently run a farm of queue workers (using redis), so would be no easy way to maintain the counts across all the servers without using redis alongside disque.
On `ADDJOB` I see there is `DELAY` and `MAXLEN` which is a good start though :+1:
Possible solutions:
1. Allow storing arbitrary data onto queues, then let the clients handle limits / any other use cases for storing queue specific data.
2. Allow storing arbitrary data on disque, same as above but not queue specific - probably better due to queues getting deleted when empty. Doesn't need to be another redis, just the basic get / set sort of stuff?
3. Native feature of disque?
Or any other suggestions? Not sure if I'm the only one with this use case though? :hear_no_evil:
Thanks,
Mike
コントリビューションガイド
このリポジトリのコントリビューションガイドは索引されていません
評価
この issue はまだ評価されていません。