beanstalkd / beanstalkd/beaneater
Weighted queues
- Dominant language
- Ruby
- Stars
- 201
- Forks
- 45
- PR merge metrics
- No merged PRs in 30d
Description
Hi there,
First, thanks for writing this excellent software - I've been using it for 10+ years and processed literally billions of jobs with it.
I recently started using weighted queues in another project which uses Sidekiq and I really enjoy not having to worry about fine turning priorities and running multiple job processors to avoid queue starvation. It's nice to get a predictable amount of processing for every queue.
Is there any way to achieve something similar with Beanstalkd? Essentially, selecting from job queues in a weighted random fashion?
I imagine I could rig up a system that uses weights to randomly select a queue, peak to see if there are jobs ready and if so, reserve a job from that queue. Rough code below:
```ruby
pipes = [{pipe: 'low_priority', weight: 1}, {pipe: 'medium_priority', weight: 2}, {pipe: 'high_priority', weight: 4}]
q_min = pipes.min_by {|q| q.dig(:weight)}.dig(:weight)
q_max = pipes.inject(0) {|r, q| r + q.dig(:weight) }
range = q_min..q_max
loop do
## Randomly select a queue based on weights and assign queue name to pipe_to_process
q_rand = Random.rand(range)
q_accumulate = 0
pipe_to_process = pipes.find do |q|
q_accumulate += q.dig(:weight)
q_accumulate >= q_rand
end
pipe = pipe_to_process.dig(:pipe)
unless beanstalk.tubes.find(pipe).peek(:ready).nil?
puts "Getting job from #{pipe}"
beanstalk.tubes.watch!(pipe)
job = beanstalk.tubes.reserve(1)
puts "Got job: #{job.id} : tube: #{pipe}"
job.release delay: 5
else
puts "No jobs in #{pipe}"
end
end
```
Not ideal to wait on a queue that might be empty ( if running multiple processors ). And when there are no jobs it thrashes between all the queues. Any thoughts on better ways to get weighted queues?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the Ruby prototype in the issue and review Beaneater's tube watch, peek, and reserve APIs. Determine how weighted selection should behave when queues are empty or multiple processors are running; done requires a maintainer-approved implementation or documented approach.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- ruby
- Domain
- backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100