Arrow: Improve initial capacity estimation for variable-width vectors
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Apache Iceberg version
main (development)
### Query engine
Spark
### Please describe the improvement
`VectorizedArrowReader` initializes variable-width vectors with:
```java
vector.setInitialCapacity(batchSize * AVERAGE_VARIABLE_WIDTH_RECORD_SIZE);
```
The single-argument Arrow API accepts a value count, not a byte count. For a batch size of 5,000 and an average-width constant of 10, this reserves offsets for 50,000 values and uses Arrow's default density of 8 bytes/value, rather than reserving 5,000 values at an estimated 10 bytes/value.
The reader already receives `ColumnChunkMetaData` in `setRowGroupInfo`, including total compressed/uncompressed sizes and value count. We should investigate using that metadata to estimate a bounded average width and initialize `BaseVariableWidthVector` with the density-aware API:
```java
setInitialCapacity(batchSize, estimatedBytesPerValue)
```
The estimate should:
- use `batchSize` as the value capacity;
- fall back to a conservative default when metadata is missing or unusable;
- account for page/dictionary overhead and avoid unbounded over-allocation;
- cover physical BINARY and logical variable-width types such as string, JSON, BSON, geometry, and geography;
- preserve safe reallocation for unusually large values.
Related: #11672 identified the value-count/byte-count mismatch but was closed automatically as stale without a fix.
### Willingness to contribute
- [X] I can contribute a fix for this improvement independently
- [ ] I would be willing to contribute a fix with guidance from the Iceberg community
- [ ] I cannot contribute a fix at this time
Contributor guide
Research direction
Start in VectorizedArrowReader, especially setRowGroupInfo, and inspect how ColumnChunkMetaData reaches variable-width vector initialization. Check the Arrow BaseVariableWidthVector density-aware API and the handling of physical BINARY plus logical variable-width types; done means bounded metadata-based estimates, a conservative fallback, and safe reallocation for large values.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100