RoaringBitmap / RoaringBitmap/RoaringFormatSpec

About memory alignment of container values

Open
#9 9 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Kaitai Struct
Stars
191
Forks
16
PR merge metrics
No merged PRs in 30d

Description

Hey @lemire,

I am currently in the process of greatly improving the deserialization algorithm of the roaring-rs library. The binary serialization format is quite good. It allows the Bitmap types to borrow a reference to the memory from which it deserializes.

However, there is a little problem, a memory alignment problem. In Rust (and other languages), it is undefined behavior to read a type that isn't well aligned in memory, e.g., reading a bitmap, aligned on 64bits, at address 33 (33 % 8 is 1, not 0, therefore unaligned).

I have two solutions to this problem in mind:

  • The easy one could be to backup on copying the bytes of a container into an aligned buffer (a Vec) if those are not valid. This is a little bit frustrating as it could happen a lot.
  • The alternative is more a question: Do you think that I could use the header offsets when serializing to force the containers to be correctly aligned, introducing some garbage bytes between containers? Or do I need to keep no space between contiguous containers?

Thank you!

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the roaring-rs deserialization algorithm and the binary format's header offsets, focusing on how borrowed container memory is aligned. Compare the proposed aligned-copy fallback with adding padding between containers. Done means the format specification documents a clear alignment decision and its serialization implications.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.