Discussion: should set FileScanConfig's batch_size to min(batch_size, limit)
Open
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
If a limit is set for FileScanConfig, I wonder if it's reasonable to also set the batch_size to min(batch_size, limit), which might reduces the number of rows being scanned and decoded.
Originally posted https://github.com/apache/arrow-rs/issues/8836
Contributor guide
Research direction
Start by tracing FileScanConfig and how its batch_size and limit are applied, then read the linked apache/arrow-rs issue 8836 for context. The issue is a discussion rather than a defined task, so completion depends on agreeing whether batch_size should be bounded by limit and documenting the resulting behavior and tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100