apache / apache/arrow-nanoarrow

Add support for reading Arrow Files (with footer)

Open
#728 1 comment 3 reactions 0 assignees View on GitHub
Dominant language
C
Stars
249
Forks
68
Avg merge
2d 21h
Merged PRs (30d)
11

Description

We have support for reading the information in the footer:

https://github.com/apache/arrow-nanoarrow/blob/bdb7d0cad6676bb93762f5e1ae6180303503457b/src/nanoarrow/ipc/decoder.c#L1312-L1345

...but this is not well integrated into the rest of the read process.

duckdb-nanoarrow implements the ability to read an arrow file as if were a stream (by skipping the ARROW1\0\0 at the start of the file and erroring if it sees any dictionaries), which we should possibly implement as part of our stream reading as well (at least as an option).

https://github.com/paleolimbot/duckdb-nanoarrow/issues/14

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the footer-reading code in src/nanoarrow/ipc/decoder.c at lines 1312-1345, then read the linked duckdb-nanoarrow issue for the proposed file-versus-stream behavior. Clarify whether stream reading should support Arrow files as an option and how dictionaries should be handled; done means the chosen behavior is integrated into the read process.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.