apache / apache/arrow-nanoarrow
Add support for reading Arrow Files (with footer)
- Dominant language
- C
- Stars
- 249
- Forks
- 68
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 11
Description
We have support for reading the information in the footer:
https://github.com/apache/arrow-nanoarrow/blob/bdb7d0cad6676bb93762f5e1ae6180303503457b/src/nanoarrow/ipc/decoder.c#L1312-L1345
...but this is not well integrated into the rest of the read process.
duckdb-nanoarrow implements the ability to read an arrow file as if were a stream (by skipping the ARROW1\0\0 at the start of the file and erroring if it sees any dictionaries), which we should possibly implement as part of our stream reading as well (at least as an option).
https://github.com/paleolimbot/duckdb-nanoarrow/issues/14
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the footer-reading code in src/nanoarrow/ipc/decoder.c at lines 1312-1345, then read the linked duckdb-nanoarrow issue for the proposed file-versus-stream behavior. Clarify whether stream reading should support Arrow files as an option and how dictionaries should be handled; done means the chosen behavior is integrated into the read process.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 38/100