apache / apache/arrow-rs

Don't use u128 for BinaryView to support alignment < 16

Open
#7,709 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Describe the bug**

The `View` inside the `BinaryView` arrays doesn't need to be represented as u128.

Representing as u128 is a problem because it requires alignment >= 16 and this is not strictly needed.

As an example, this is how polars-arrow defines it and it specifically mentions u128 isn't used to avoid having a large alignment requirement.

```rust
#[repr(C)]
pub struct View {
/// The length of the string/bytes.
pub length: u32,
/// First 4 bytes of string/bytes data.
pub prefix: u32,
/// The buffer index.
pub buffer_idx: u32,
/// The offset into the buffer.
pub offset: u32,
}
```

https://docs.rs/polars-arrow/latest/src/polars_arrow/array/binview/view.rs.html

**Expected behavior**

alignment >= 16 requirement should be removed for BinaryView arrays when doing FFI between other arrow libraries and arrow-rs

**Additional context**

The arrow spec doesn't define the view struct to have u128 representation, it defines it like a regular struct that would have alignment requirement 4 in a c representation.

https://arrow.apache.org/docs/format/Columnar.html#variable-size-binary-view-layout

Contributor guide

Open the contributing guide

Research direction

Start by locating the BinaryView arrays, their View representation, and the FFI conversion paths in arrow-rs. Compare them with the polars-arrow View layout and the Arrow variable-size binary view specification; done means BinaryView FFI no longer requires alignment >= 16.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.