apache / apache/arrow-java

[Java] Support reading a subset of fields from an IPC file or stream

Open
#242 0 comments 1 reaction 0 assignees View on GitHub
Type: enhancement
Dominant language
Java
Stars
94
Forks
152
Avg merge
3d 16h
Merged PRs (30d)
11

Description

C++ supports `IpcReadOptions.included_fields` which lets you load a subset of (top-level) fields from an IPC file or stream, potentially saving on I/O costs. It would be useful to support this in Java as well. Some refactoring would be required since MessageSerializer currently reads record batch messages in as a whole, and it would be good to quantify how much of a benefit this provides in different scenarios.

**Reporter**: [David Li](https://issues.apache.org/jira/browse/ARROW-17914) / @lidavidm

**Note**: *This issue was originally created as [ARROW-17914](https://issues.apache.org/jira/browse/ARROW-17914). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start with MessageSerializer and trace how record batch messages are currently read from IPC files and streams. Compare the desired behavior with C++ IpcReadOptions.included_fields, then define how top-level field selection should work and quantify the I/O benefit across different scenarios.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.