JuliaCollections / JuliaCollections/DataStructures.jl

Priority Queue implemented as Bucket Queue for Integer weights (4x speed)

Open
#504 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Julia
Stars
745
Forks
261
PR merge metrics
No merged PRs in 30d

Description

I suggest adding the bucket queue algorithm, which is very fast for small integer weights.
A lot of problems don't require a perfect 'float' priority queue but can use a rounded integer version that is multiple times faster by using a fixed range of possible integer weights.

For my application its runtime is 4s compared to 28s of the standard PriorityQueue{Int, Int}.

I would create a pull request, but I'm not sure how to best integrate it in the existing PriorityQueue. The underlying data structure and algorithm is different, so the whole struct would need a second version, I guess.

My code for the bucket queue is very short:

mutable struct PQueue{T}
    min::Int
    nbins::Int
    content::Vector{Vector{T}}
end

# initialize the buckets
PQueue{T}(nbins) where T = PQueue(nbins + 1, nbins, [Vector{T}() for _ in 1:nbins])

Base.isempty(q::PQueue) = q.min > q.nbins

function Base.push!(q::PQueue, item, weight)
    push!(q.content[weight], item)
    q.min = min(q.min, weight)
    return q
end

function Base.pop!(q::PQueue)
    elem = pop!(q.content[q.min])
    # increase smallestbin, if elem was last in bin
    while q.min ≤ q.nbins && isempty(q.content[q.min])
        q.min += 1
    end
    return elem
end

And a script showing a 4x higher speed in a use case, where the queue is growing:

using DataStructures
include("priorityqueue.jl")

nbins = 255 # small number
ndata = 10000000
randdata = rand(1:nbins, ndata)

function pq_datastructures()
    pq = PriorityQueue{Int, Int}()
    for i in 1:ndata
        enqueue!(pq, i, randdata[i])
        if iseven(i)
            dequeue!(pq)
        end
    end
end

function pq_bucketqueue()
    pq = PQueue{Int}(nbins)
    for i in 1:ndata
        push!(pq, i, randdata[i])
        if iseven(i)
            pop!(pq)
        end
    end
end

pq_datastructures()
pq_bucketqueue()

@time pq_datastructures()
@time pq_datastructures()

@time pq_bucketqueue()
@time pq_bucketqueue()

# output
  6.768914 seconds (30.00 M allocations: 920.507 MiB, 4.74% gc time)
  6.738860 seconds (30.00 M allocations: 920.507 MiB, 5.00% gc time)

  1.551067 seconds (35.00 M allocations: 814.256 MiB, 13.11% gc time)
  1.496581 seconds (35.00 M allocations: 814.256 MiB, 13.03% gc time)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by comparing the proposed PQueue implementation with the existing PriorityQueue API and the included benchmark script. Determine whether the bucket queue should be a separate type or integrated as a second implementation, then benchmark the chosen design against PriorityQueue for bounded integer weights. Done means the queue supports the intended operations and demonstrates the reported performance improvement.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.