[C++] Acero/dataset relies on ExecBatch::ToRecordBatch truncating excess columns
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
As found while working on ARROW-18004: the dataset scanner and the Acero engine rely on `ExecBatch::ToRecordBatch` returning successfully when the given schema has fewer fields than the ExecBatch has columns.
This apparently allows to implicitly drop the dataset-added columns (`kAugmentedFields` in `arrow/dataset/scanner.cc`) from a scan's final result.
However, it seems wrong and brittle to do this implicitly at the `ExecBatch::ToRecordBatch` level (hiding potential errors). Instead, it should probably be done explicitly inside Acero/dataset.
**Reporter**: [Antoine Pitrou](https://issues.apache.org/jira/browse/ARROW-18037) / @pitrou
#### Related issues:
- [[C++] ExecBatch conversion to RecordBatch may go out of bounds](https://github.com/apache/arrow/issues/33208) (is related to)
**Note**: *This issue was originally created as [ARROW-18037](https://issues.apache.org/jira/browse/ARROW-18037). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start by reading ExecBatch::ToRecordBatch and the dataset scanner code at arrow/dataset/scanner.cc, then trace the Acero/dataset paths that depend on schemas with fewer fields. Compare the behavior with related issue #33208. Done means column dropping is explicit in Acero or dataset rather than implicitly accepted by ExecBatch::ToRecordBatch, with coverage for the affected scan or execution path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- data-engineering
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100