apache / apache/iceberg

Arrow: Improve initial capacity estimation for variable-width vectors

Open
#17,305 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Apache Iceberg version

main (development)

### Query engine

Spark

### Please describe the improvement

`VectorizedArrowReader` initializes variable-width vectors with:

```java
vector.setInitialCapacity(batchSize * AVERAGE_VARIABLE_WIDTH_RECORD_SIZE);
```

The single-argument Arrow API accepts a value count, not a byte count. For a batch size of 5,000 and an average-width constant of 10, this reserves offsets for 50,000 values and uses Arrow's default density of 8 bytes/value, rather than reserving 5,000 values at an estimated 10 bytes/value.

The reader already receives `ColumnChunkMetaData` in `setRowGroupInfo`, including total compressed/uncompressed sizes and value count. We should investigate using that metadata to estimate a bounded average width and initialize `BaseVariableWidthVector` with the density-aware API:

```java
setInitialCapacity(batchSize, estimatedBytesPerValue)
```

The estimate should:

- use `batchSize` as the value capacity;
- fall back to a conservative default when metadata is missing or unusable;
- account for page/dictionary overhead and avoid unbounded over-allocation;
- cover physical BINARY and logical variable-width types such as string, JSON, BSON, geometry, and geography;
- preserve safe reallocation for unusually large values.

Related: #11672 identified the value-count/byte-count mismatch but was closed automatically as stale without a fix.

### Willingness to contribute

- [X] I can contribute a fix for this improvement independently
- [ ] I would be willing to contribute a fix with guidance from the Iceberg community
- [ ] I cannot contribute a fix at this time

Contributor guide

Open the contributing guide

Research direction

Start in VectorizedArrowReader, especially setRowGroupInfo, and inspect how ColumnChunkMetaData reaches variable-width vector initialization. Check the Arrow BaseVariableWidthVector density-aware API and the handling of physical BINARY plus logical variable-width types; done means bounded metadata-based estimates, a conservative fallback, and safe reallocation for large values.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.