[C++] Try and maintain input shape in Acero
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
Data is scanned in large chunks based on the format. For example, CSV scans chunks based on a chunk_size while parquet scans entire row groups.
Then, upon entry into Acero, these chunks are sliced into morsels (~L3 size) for parallelism and batches (~L1-L2 size) for cache efficient processing.
However, the way it is currently done, means that the output of Acero is a stream of tiny batches. This is somewhat undesirable in many cases.
For example, if a pyarrow user calls pq.read_table they might expect to get one batch per row group. If they were to turn around and write out that table to a new parquet file then either they end up with a non-ideal parquet file (tiny row groups) or they are forced to concatenate the batches (which is an allocation + copy).
Even if the user is doing their own streaming processing (e.g. in pyarrow) these small batch sizes are undesirable as the overhead of python means that streaming processing should be done in larger batches.
Instead, there should be a configurable max_batch_size, independent of row group size and morsel size, which is configurable, and quite large by default (1Mi or 64Mi rows). This control exists for users that want to do their own streaming processing and need to be able to tune for RAM usage.
Acero will read in data based on the format, as it does today (e.g. CSV chunk size, row group size). If the source data is very large (bigger than max_batch_size) it will be sliced. From that point on, any morsels or batches should simply be views into this larger output batch. For example, when doing a projection to add a new column, we should allocate a max_batch_size array and then populate it over many runs of the project node.
**Reporter**: [Weston Pace](https://issues.apache.org/jira/browse/ARROW-17593) / @westonpace
**Note**: *This issue was originally created as [ARROW-17593](https://issues.apache.org/jira/browse/ARROW-17593). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
No file or test is named in the issue. Start by tracing Acero's scan-to-morsel and batch pipeline, then review how format chunking and node outputs are configured. Done means a configurable max_batch_size preserves larger output batches while allowing oversized sources to be sliced and downstream morsels to remain views.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100