apache / apache/arrow

[C++][Datasets] Investigate sub-batch IPC reads

Open
#30,895 3 comments 0 reactions 0 assignees View on GitHub
Component: C++ Type: task
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

When scanning an IPC file the finest resolution we can read currently is a record batch. Often we are processing relatively small slices of that batch in an iterative fashion. This means we sometimes have to read in and hold a huge batch of memory while we slice off small pieces of it.

For example, if a user creates an IPC file with 1 record batch with 50 million rows and we want to process it in batches of 64K rows we have to first read the entire 50 million rows in memory and then slice off the 64K sub-batches.

We should be able to create a sub-batch reader (although this will be more complicated in the future with things like RLE columns) which can slice small pieces of the batch off the disk instead of reading the entire batch into memory first.

**Reporter**: [Weston Pace](https://issues.apache.org/jira/browse/ARROW-15413) / @westonpace

**Note**: *This issue was originally created as [ARROW-15413](https://issues.apache.org/jira/browse/ARROW-15413). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Investigate the existing IPC scanning and record-batch reading path; the issue does not name specific files, tests, or entry points. Define how a sub-batch reader should read small row ranges without loading an entire large batch, and determine how complications such as RLE columns affect the design.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.