Don't use u128 for BinaryView to support alignment < 16
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
**Describe the bug**
The `View` inside the `BinaryView` arrays doesn't need to be represented as u128.
Representing as u128 is a problem because it requires alignment >= 16 and this is not strictly needed.
As an example, this is how polars-arrow defines it and it specifically mentions u128 isn't used to avoid having a large alignment requirement.
```rust
#[repr(C)]
pub struct View {
/// The length of the string/bytes.
pub length: u32,
/// First 4 bytes of string/bytes data.
pub prefix: u32,
/// The buffer index.
pub buffer_idx: u32,
/// The offset into the buffer.
pub offset: u32,
}
```
https://docs.rs/polars-arrow/latest/src/polars_arrow/array/binview/view.rs.html
**Expected behavior**
alignment >= 16 requirement should be removed for BinaryView arrays when doing FFI between other arrow libraries and arrow-rs
**Additional context**
The arrow spec doesn't define the view struct to have u128 representation, it defines it like a regular struct that would have alignment requirement 4 in a c representation.
https://arrow.apache.org/docs/format/Columnar.html#variable-size-binary-view-layout
Contributor guide
Research direction
Start by locating the BinaryView arrays, their View representation, and the FFI conversion paths in arrow-rs. Compare them with the polars-arrow View layout and the Arrow variable-size binary view specification; done means BinaryView FFI no longer requires alignment >= 16.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100