apache / apache/arrow-java

[Java] BaseVariableWidthVector only supports 1 GiB through safe interfaces

未關閉
#220 1 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
Type: bug
主要語言
Java
星號
94
分支
152
平均合併
3 天 16 小時
30 天內合併 PR
11

描述

### Describe the bug, including details regarding any error messages, version, and platform.

https://github.com/apache/arrow/pull/13815 fixed an issue with a misleading error caused by going over the 2GiB limit of a variable width vector, but it also introduced a 1 GiB limit when using the safe interfaces. Basically whenever you try to add data beyond 1 GiB, the vector will try to double itself to the next power of two, which would be 2147483648, which is greater than Integer.MAX_VALUE which is 2147483647, thus throwing a OversizedAllocationException. This effectively limits the total size allowed in the buffer to 1 GiB, even though it should only really be bound by the max offset being less than Integer.MAX_VALUE.

See https://github.com/apache/spark/pull/39572#issuecomment-1383195213 and the comment above it for how I could recreate the issue.

### Component(s)

Java

貢獻指南

開啟貢獻指南

研究方向

使用連結的 Apache Spark 討論中描述的情境重現 1 GiB 失敗,然後追蹤 BaseVariableWidthVector 及其安全介面中的容量成長。當可變寬度資料能夠成長超過 1 GiB,而不會嘗試進行無效的 2 GiB 配置,同時仍遵守 Integer.MAX_VALUE 位移量限制時,修正即告完成;為此邊界新增回歸測試。

由索引模型根據 Issue 內容生成。

評估

技術堆疊
java
領域
data-engineering
Issue 類型
缺陷
難度
4/5
預估耗時
3-5 天
活躍度
停滯
描述清晰度
基本清楚
新手友好度
45/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。