apache / apache/iceberg

BaseFile.splitOffsets() allocates a new List on every call

Open
#15,622 1 comment 0 reactions 0 assignees View on GitHub
improvement stale
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Feature Request / Improvement

The reason why we care about this is when we have parquet manifests we cannot re-use the immutable list returned by the "get" method from base file. That means we leak an object for every manifest. Not a huge deal but we should probably do something there.

--

`BaseFile` stores split offsets internally as a `long[]`, but [`splitOffsets()`](https://github.com/apache/iceberg/blob/fb81fcf3827fa14302507da04820f5858f834bfd/core/src/main/java/org/apache/iceberg/BaseFile.java#L532-L538) wraps it in a new `List` via `ArrayUtil.toUnmodifiableLongList` on every invocation. When file metadata is being read and rewritten (e.g., during manifest rewriting or format conversion), this means each entry needlessly allocates a list that is immediately consumed and discarded.

Other fields like `partitionData` are stored and returned as-is. Split offsets could similarly cache or reuse the `List` representation, or callers within the `core` module could use the existing package-private [`splitOffsetArray()`](https://github.com/apache/iceberg/blob/fb81fcf3827fa14302507da04820f5858f834bfd/core/src/main/java/org/apache/iceberg/BaseFile.java#L540-L546) to pass the raw `long[]` through without conversion.

### Query engine

None

### Willingness to contribute

- [x] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.