apache / apache/arrow-java

[Java][Docs] Document the use of the batchSize argument in Dataset ScanOptions

Open
#303 1 comment 0 reactions 0 assignees View on GitHub
Type: enhancement
Dominant language
Java
Stars
94
Forks
152
Avg merge
3d 16h
Merged PRs (30d)
11

Description

Several ScanOptions methods take a batchSize argument as shown: 

`public ScanOptions(long batchSize) {`
`    this(batchSize, Optional.empty());`
``}

Since the scanner reads one ArrowRecordBatch per load invocation, setting the parameter to a size larger than the RecordBatch has no effect. It only works when it's smaller than the number of rows in the RecordBatch, (i.e., the number or records read is equal to min(batchSize, recordBatch rowCount), potentially leading to some confusion. 

**Reporter**: [Larry White](https://issues.apache.org/jira/browse/ARROW-17346) / @lwhite1

**Note**: *This issue was originally created as [ARROW-17346](https://issues.apache.org/jira/browse/ARROW-17346). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start with the Dataset ScanOptions methods that accept batchSize and trace how the scanner loads one ArrowRecordBatch per invocation. Document that values larger than the RecordBatch row count have no additional effect, while smaller values limit records to min(batchSize, recordBatch rowCount); done means the ScanOptions documentation makes this behavior clear.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
documentation
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.