apache / apache/arrow

[C++][Dataset] Change scanner readahead limits to be based on bytes instead of number of batches

Open
#30,191 2 comments 0 reactions 0 assignees View on GitHub
Component: C++ Priority: Critical Status: needs champion Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

In the scanner readahead is controlled by "batch_readahead" and "fragment_readahead" (both specified in the scan options).  This was mainly motivated on my work with CSV and the defaults of 32 and 8 will cause the scanner to buffer ~256MB of data (given the default block size of 1MB).

For parquet / IPC this would mean we are buffering 256 row groups which is entirely too high.

Rather than make users figure out complex parameters we should have a single readahead limit that is specified in bytes.

This will be "best effort".  I'm not suggest we support partial reads of row groups / record batches so if the limit is set very small we still might end up with more in RAM just because we can only load entire row groups.

**Reporter**: [Weston Pace](https://issues.apache.org/jira/browse/ARROW-14648) / @westonpace
#### Related issues:
- [[C++] Change dataset readahead to be based on available RAM/CPU instead of fixed constants/options](https://github.com/apache/arrow/issues/27859) (is duplicated by)
- [[C++] Improve performance of parquet readahead](https://github.com/apache/arrow/issues/31683) (is related to)
- [[C++][R]Opening a multi-file dataset and writing a re-partitioned version of it fails](https://github.com/apache/arrow/issues/18944) (is depended upon by)
- [[C++][Datasets] Improve memory usage of datasets](https://github.com/apache/arrow/issues/30893) (is depended upon by)

**Note**: *This issue was originally created as [ARROW-14648](https://issues.apache.org/jira/browse/ARROW-14648). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start with the C++ dataset scanner and its scan options, tracing how batch_readahead and fragment_readahead control buffering. Review the related memory-usage and performance issues before deciding the scope. Done means readahead is governed by a best-effort byte limit while still allowing complete row groups or record batches.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data-engineering, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.