Arrow: ColumnVector.getArrowVector() leaks direct memory for dictionary encoded columns
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Apache Iceberg version
1.11.0 (latest release)
### Query engine
Other
### Please describe the bug 🐞
### Summary
Every call to `ColumnVector.getArrowVector()` on a dictionary encoded column allocates a new decoded `FieldVector` that is never released. Since `ColumnarBatch.createVectorSchemaRootFromVectors()` calls it for every column of every batch, a scan over dict-encoded data leaks one vector per batch per dict-encoded
column. The memory survives a fully drained and fully closed scan..
`ColumnVector.getArrowVector()` routes dict-encoded columns through [`DictEncodedArrowConverter.toArrowVector()`](https://github.com/apache/iceberg/blob/main/arrow/src/main/java/org/apache/iceberg/arrow/DictEncodedArrowConverter.java#L51-L85) which allocates and populates a new vector from the reader's allocator on each call. And [`ColumnVector.close()`](https://github.com/apache/iceberg/blob/main/arrow/src/main/java/org/apache/iceberg/arrow/vectorized/ColumnVector.java#L85-L88) closes only the accessor , `BaseBatchReader.closeVectors()` closes only the holders' vectors
The leak goes undetected by Arrow's allocator leak checking because the `VectorizedReadBuilder` child allocator the memory is charged to is itself never closed. so the check never runs.
### Reproduction
Write a Parquet file whose string column has few distinct values (so pages are dict-encoded), scan it with `VectorizedTableScanIterable`, call `createVectorSchemaRootFromVectors()` on each batch, close everything, and compare `ArrowAllocation.rootAllocator().getAllocatedMemory()` against the pre scan baseline taken after creating the dictionaryEncodedTable. The allocated memory never comes back down it stays higher by exactly the size of the decoded vectors (33,280 bytes for 100 rows / 1 batch / 1 string column on current main). Running the same scan without calling createVectorSchemaRootFromVectors() returns to baseline, so the conversion is what's leaking.
### Related issues
- #13937 — `VectorizedReadBuilder` child allocator never closed (masks this leak)
### Proposed fix
Make the reader own the decoded vectors.
- `ColumnVector` materializes the decoded vector once per batch, instead of allocating on every `getArrowVector()`.
- `ArrowBatchReader` keeps a reference to the `ColumnVector`s it hands out and releases their decoded vectors at the start of the next `read()` and in `close()`.
- Only vectors allocated by the dictionary decoding are released.
This also matches the documented contract ("the arrow vectors are owned by the reader") without requiring callers to close batches.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
Contributor guide
Research direction
Start by reading ColumnVector.java, DictEncodedArrowConverter.java, ArrowBatchReader, and the ColumnarBatch.createVectorSchemaRootFromVectors() entry point to trace decoded-vector ownership. Run the described dictionary-encoded scan and compare ArrowAllocation.rootAllocator().getAllocatedMemory() before and after closing; done means memory returns to the pre-scan baseline while non-dictionary vectors remain correctly managed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 64/100