apache / apache/iceberg

Arrow: ColumnVector.getArrowVector() leaks direct memory for dictionary encoded columns

Open
#17,722 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Apache Iceberg version

1.11.0 (latest release)

### Query engine

Other

### Please describe the bug 🐞

### Summary
Every call to `ColumnVector.getArrowVector()` on a dictionary encoded column allocates a new decoded `FieldVector` that is never released. Since `ColumnarBatch.createVectorSchemaRootFromVectors()` calls it for every column of every batch, a scan over dict-encoded data leaks one vector per batch per dict-encoded
column. The memory survives a fully drained and fully closed scan..

`ColumnVector.getArrowVector()` routes dict-encoded columns through [`DictEncodedArrowConverter.toArrowVector()`](https://github.com/apache/iceberg/blob/main/arrow/src/main/java/org/apache/iceberg/arrow/DictEncodedArrowConverter.java#L51-L85) which allocates and populates a new vector from the reader's allocator on each call. And [`ColumnVector.close()`](https://github.com/apache/iceberg/blob/main/arrow/src/main/java/org/apache/iceberg/arrow/vectorized/ColumnVector.java#L85-L88) closes only the accessor , `BaseBatchReader.closeVectors()` closes only the holders' vectors

The leak goes undetected by Arrow's allocator leak checking because the `VectorizedReadBuilder` child allocator the memory is charged to is itself never closed. so the check never runs.

### Reproduction
Write a Parquet file whose string column has few distinct values (so pages are dict-encoded), scan it with `VectorizedTableScanIterable`, call `createVectorSchemaRootFromVectors()` on each batch, close everything, and compare `ArrowAllocation.rootAllocator().getAllocatedMemory()` against the pre scan baseline taken after creating the dictionaryEncodedTable. The allocated memory never comes back down it stays higher by exactly the size of the decoded vectors (33,280 bytes for 100 rows / 1 batch / 1 string column on current main). Running the same scan without calling createVectorSchemaRootFromVectors() returns to baseline, so the conversion is what's leaking.

### Related issues
- #13937 — `VectorizedReadBuilder` child allocator never closed (masks this leak)

### Proposed fix
Make the reader own the decoded vectors.
- `ColumnVector` materializes the decoded vector once per batch, instead of allocating on every `getArrowVector()`.
- `ArrowBatchReader` keeps a reference to the `ColumnVector`s it hands out and releases their decoded vectors at the start of the next `read()` and in `close()`.
- Only vectors allocated by the dictionary decoding are released.

This also matches the documented contract ("the arrow vectors are owned by the reader") without requiring callers to close batches.

### Willingness to contribute

- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time

Contributor guide

Open the contributing guide

Research direction

Start by reading ColumnVector.java, DictEncodedArrowConverter.java, ArrowBatchReader, and the ColumnarBatch.createVectorSchemaRootFromVectors() entry point to trace decoded-vector ownership. Run the described dictionary-encoded scan and compare ArrowAllocation.rootAllocator().getAllocatedMemory() before and after closing; done means memory returns to the pre-scan baseline while non-dictionary vectors remain correctly managed.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
64/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.