[Java] BaseVariableWidthVector only supports 1 GiB through safe interfaces
- Dominant language
- Java
- Stars
- 94
- Forks
- 152
- Avg merge
- 3d 16h
- Merged PRs (30d)
- 11
Description
### Describe the bug, including details regarding any error messages, version, and platform.
https://github.com/apache/arrow/pull/13815 fixed an issue with a misleading error caused by going over the 2GiB limit of a variable width vector, but it also introduced a 1 GiB limit when using the safe interfaces. Basically whenever you try to add data beyond 1 GiB, the vector will try to double itself to the next power of two, which would be 2147483648, which is greater than Integer.MAX_VALUE which is 2147483647, thus throwing a OversizedAllocationException. This effectively limits the total size allowed in the buffer to 1 GiB, even though it should only really be bound by the max offset being less than Integer.MAX_VALUE.
See https://github.com/apache/spark/pull/39572#issuecomment-1383195213 and the comment above it for how I could recreate the issue.
### Component(s)
Java
Contributor guide
Research direction
Reproduce the 1 GiB failure using the scenario described in the linked Apache Spark discussion, then trace capacity growth in BaseVariableWidthVector and its safe interfaces. The fix is done when variable-width data can grow beyond 1 GiB without attempting an invalid 2 GiB allocation, while still respecting the Integer.MAX_VALUE offset limit; add a regression test for this boundary.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100