apache / apache/arrow-rs

IPC code writes files which do not include a valid stream

Open
#6,311 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Describe the bug**

The format guarantees that each IPC file [embeds a valid IPC stream](https://github.com/apache/arrow/blob/bde6ac57ad943f5938506359de1b13fb85b4f8ea/docs/source/format/Columnar.rst) in order to allow readers to ignore the Footer, skip the file's leading magic, and reuse a stream reader.

However, when writing IPC files arrow-rs aligns the encapsulated flatbuffers Messages to 64 byte boundaries instead of 8 bytes. This can leave gaps of padding bytes between the Messages which a stream reader would not know to skip.

**To Reproduce**

https://github.com/apache/arrow/pull/43834 adds validation of the embedded stream to arrow-c++'s `arrow-json-integration-test`. Running this against IPC files written by arrow-rs [raises an error](https://github.com/apache/arrow/actions/runs/10565413633/job/29269867382?pr=43834#step:9:27320)

```shell-session
$ arrow-json-integration-test -arrow datetime.arrow -json datetime.json -integration -mode VALIDATE
Error message: Invalid: Tried reading schema message, was null or length 0
```

**Expected behavior**

A stream reader should be able to read an IPC file by skipping the first 8 bytes.

**Additional context**

This was originally introduced in https://github.com/apache/arrow-rs/commit/eddef43d1cb46c1287da187ea1d86b0e1dc35a13 which added alignment to address new requirements around `i128`. However the alignment should not be applied to flatbuffers Messages; apart from the above issue I think there's no SIMD or other advantage to aligning those to more than 8 bytes. Body buffers can of course still be padded and aligned freely.

This was discovered while adding IPC file reading to nanoarrow; we were trying to defer reading Footers for a follow up and discovered that the go, rust, and javascript implementations don't embed a valid stream. Most readers have not noticed because offsets and schemas are more efficiently read from a Footer, and once acquired obviate sequential stream-style reading.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the arrow-rs IPC file-writer alignment logic introduced by commit eddef43, then reproduce the failure with arrow-json-integration-test using datetime.arrow and datetime.json in VALIDATE mode. Done means an IPC file written by arrow-rs can be read as an embedded stream after skipping its first 8 bytes, while body buffers retain valid padding.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.