apache / apache/arrow

[C++] Scanner slicing large row groups leads to inefficient RAM usage

Open
#20,473 4 comments 0 reactions 0 assignees View on GitHub
Component: C++ Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

As an example, consider a 4GB parquet file with 1 giant row group. At the moment it is inevitable that we read this in as one large 4GB record batch (there are other JIRAs for sub-row-group reads which, if implemented, would obsolete this one).

We then slice off pieces of that 4GB parquet file for processing:

```

next_batch = current.slice(0, batch_size)
current = current.slice(batch_size)
```

However, even though `current` is shrinking each time, it always references the entire data (slicing doesn't allow memory to be freed). We may want to investigate alternative strategies here so that we can free up memory when we are done processing it.

**Reporter**: [Weston Pace](https://issues.apache.org/jira/browse/ARROW-18160) / @westonpace

**Note**: *This issue was originally created as [ARROW-18160](https://issues.apache.org/jira/browse/ARROW-18160). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by tracing the scanner's row-group batching and the current.slice usage described in the issue. Compare possible approaches for releasing memory after each processed slice, and consider the 4GB single-row-group example as the completion criterion while preserving scanner behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.