apache / apache/arrow-rs

ByteView should be using signed types?

Open
#6,172 3 comments 4 reactions 0 assignees View on GitHub
question
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Which part is this question about**

Related to the continuation of StringView/BinaryView support in #6163

In `arrow_data`, the `ByteView` type is used to encapsulate this structure from the spec:

![image](https://github.com/user-attachments/assets/8123beca-b2ab-4431-aab6-9e916896cfa2)

Notably, the spec dictates that all of these values must be signed integers. However, we're using u32.

The arrow-rs builder for GenericByteViewArray doesn't seem to have any range checks on the block, offset and len values for the view structure, which means, _I think_, you can happily construct a StringView array with arrow-rs, and then attempt to pass it to PyArrow or Java over IPC and have it fail at runtime.

**Describe your question**

Should we either

1. be using i32 instead of u32 internally
2. be adding constraints on the builder methods to ensure that we don't allow adding strings > 2GB
3. Has someone noticed this before and addressed it and it's not actually a problem

Contributor guide

Open the contributing guide

Research direction

Start with the arrow_data ByteView and GenericByteViewArray builder APIs, then compare their u32 fields and range behavior with the Arrow view specification and the described IPC consumers. Done means determining whether the representation or builder constraints are incorrect and documenting or implementing the agreed compatibility fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.