apache / apache/arrow-java

[Java] BaseVariableWidthVector only supports 1 GiB through safe interfaces

未关闭
#220 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
Type: bug
主要语言
Java
星标
94
派生
152
平均合并
3 天 16 小时
30 天内合并 PR
11

描述

### Describe the bug, including details regarding any error messages, version, and platform.

https://github.com/apache/arrow/pull/13815 fixed an issue with a misleading error caused by going over the 2GiB limit of a variable width vector, but it also introduced a 1 GiB limit when using the safe interfaces. Basically whenever you try to add data beyond 1 GiB, the vector will try to double itself to the next power of two, which would be 2147483648, which is greater than Integer.MAX_VALUE which is 2147483647, thus throwing a OversizedAllocationException. This effectively limits the total size allowed in the buffer to 1 GiB, even though it should only really be bound by the max offset being less than Integer.MAX_VALUE.

See https://github.com/apache/spark/pull/39572#issuecomment-1383195213 and the comment above it for how I could recreate the issue.

### Component(s)

Java

贡献指南

打开贡献指南

调研方向

使用链接的 Apache Spark 讨论中描述的场景重现 1 GiB 故障,然后跟踪 BaseVariableWidthVector 及其安全接口中的容量增长。当可变宽度数据能够增长到超过 1 GiB,而不会尝试进行无效的 2 GiB 分配,同时仍遵守 Integer.MAX_VALUE 偏移量限制时,修复即告完成;为此边界添加回归测试。

由索引模型根据 Issue 内容生成。

评估

技术栈
java
领域
data-engineering
Issue 类型
缺陷
难度
4/5
预计耗时
3-5 天
活跃度
停滞
描述清晰度
基本清楚
新手友好度
45/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。