[Java] BaseVariableWidthVector only supports 1 GiB through safe interfaces
- 主要语言
- Java
- 星标
- 94
- 派生
- 152
- 平均合并
- 3 天 16 小时
- 30 天内合并 PR
- 11
描述
### Describe the bug, including details regarding any error messages, version, and platform.
https://github.com/apache/arrow/pull/13815 fixed an issue with a misleading error caused by going over the 2GiB limit of a variable width vector, but it also introduced a 1 GiB limit when using the safe interfaces. Basically whenever you try to add data beyond 1 GiB, the vector will try to double itself to the next power of two, which would be 2147483648, which is greater than Integer.MAX_VALUE which is 2147483647, thus throwing a OversizedAllocationException. This effectively limits the total size allowed in the buffer to 1 GiB, even though it should only really be bound by the max offset being less than Integer.MAX_VALUE.
See https://github.com/apache/spark/pull/39572#issuecomment-1383195213 and the comment above it for how I could recreate the issue.
### Component(s)
Java
贡献指南
调研方向
使用链接的 Apache Spark 讨论中描述的场景重现 1 GiB 故障,然后跟踪 BaseVariableWidthVector 及其安全接口中的容量增长。当可变宽度数据能够增长到超过 1 GiB,而不会尝试进行无效的 2 GiB 分配,同时仍遵守 Integer.MAX_VALUE 偏移量限制时,修复即告完成;为此边界添加回归测试。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- java
- 领域
- data-engineering
- Issue 类型
- 缺陷
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 活跃度
- 停滞
- 描述清晰度
- 基本清楚
- 新手友好度
- 45/100