RoaringBitmap / RoaringBitmap/roaring-rs
Hand tune array-bitset aggregates
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 957
- Forks
- 119
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 1
Description
arXiv:1709.07821 Roaring Bitmaps: Implementation of an Optimized Software Library (Section 4.1.1) (Section 3.2)
Cardinality tracking was added in #127, however we're leaving some perf on the floor.
We also find that if we use bit-manipulation instructions and hand-tuned assembly, we can roughly
double the speed at which we can change bits in a bitset, compared to compiler-generated machine
code with all optimization flags activated. It is true not only when setting bits, but also when clearing
or flipping them—in which cases we need to use the btr and btc instructions instead. Informally,
we tested various C compilers and could not find any that could perform nearly as well as hand-tuned
assembly code. This may, of course, change in the future.
Planning
- Do we want to implement this optimization? 2x is a huge speed up, but only on x86
- The compiler is sufficiently smart to use
btsbut notsbb, perhaps we can write rust that compiles to the asm we want- https://doc.rust-lang.org/std/primitive.u32.html#method.borrowing_sub
- Portable, but it may compile to more / higher latency μops on some architectures, we should be careful this is not a pessimization for them
- https://doc.rust-lang.org/core/arch/x86_64/fn._subborrow_u64.html
- Not portable, but that may be desirable (see above). Available in stable rust.
- https://doc.rust-lang.org/std/primitive.u32.html#method.borrowing_sub
- If not there's always the
asm!macro (nightly only), or if want it in stable then we can write a small C lib for the asm and link it
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the cardinality tracking added in #127 and compare the Rust u32 borrowing_sub and core::arch::x86_64::_subborrow_u64 options against CRoaring's src/bitset_util.c. Done would require a confirmed optimization choice and evidence that it improves bitset aggregate performance without penalizing other architectures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, rust
- Domain
- performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100