IndexOutOfBoundsException when loading compressed IPC format
- 主要語言
- Java
- 星號
- 94
- 分支
- 152
- 平均合併
- 3 天 16 小時
- 30 天內合併 PR
- 11
描述
I encountered this bug when I loaded a dataframe stored in the Arrow IPC format.
```java
// Java Code from "Apache Arrow Java Cookbook"
File file = new File("example.arrow");
try (
BufferAllocator rootAllocator = new RootAllocator();
FileInputStream fileInputStream = new FileInputStream(file);
ArrowFileReader reader = new ArrowFileReader(fileInputStream.getChannel(), rootAllocator)
) {
System.out.println("Record batches in file: " + reader.getRecordBlocks().size());
for (ArrowBlock arrowBlock : reader.getRecordBlocks()) {
reader.loadRecordBatch(arrowBlock);
VectorSchemaRoot vectorSchemaRootRecover = reader.getVectorSchemaRoot();
System.out.print(vectorSchemaRootRecover.contentToTSVString());
}
} catch (IOException e) {
e.printStackTrace();
}
```
Call stack:
```
Exception in thread "main" java.lang.IndexOutOfBoundsException: index: 0, length: 2048 (expected: range(0, 2024))
at org.apache.arrow.memory.ArrowBuf.checkIndex(ArrowBuf.java:701)
at org.apache.arrow.memory.ArrowBuf.setBytes(ArrowBuf.java:955)
at org.apache.arrow.vector.BaseFixedWidthVector.reAlloc(BaseFixedWidthVector.java:451)
at org.apache.arrow.vector.BaseFixedWidthVector.setValueCount(BaseFixedWidthVector.java:732)
at org.apache.arrow.vector.VectorSchemaRoot.setRowCount(VectorSchemaRoot.java:240)
at org.apache.arrow.vector.VectorLoader.load(VectorLoader.java:86)
at org.apache.arrow.vector.ipc.ArrowReader.loadRecordBatch(ArrowReader.java:220)
at org.apache.arrow.vector.ipc.ArrowFileReader.loadNextBatch(ArrowFileReader.java:166)
at org.apache.arrow.vector.ipc.ArrowFileReader.loadRecordBatch(ArrowFileReader.java:197)
```
This bug can be reproduced by a simple dataframe created by pandas:
```java
pd.DataFrame({'a': range(10000)}).to_feather('example.arrow')
```
Pandas compresses the dataframe by default. If the compression is turned off, Java can load the dataframe. Thus, I guess the bounds checking code is buggy when loading compressed file.
That dataframe can be loaded in polars, pandas and pyarrow, so it's unlikely to be a pandas bug.
**Environment**: Linux and Windows.
Apache Arrow Java version: 10.0.0, 9.0.0, 4.0.1.
Pandas 1.4.2 using pyarrow 8.0.0 (anaconda3-2022.05)
**Reporter**: [Georeth Zhou](https://issues.apache.org/jira/browse/ARROW-18198)
**Note**: *This issue was originally created as [ARROW-18198](https://issues.apache.org/jira/browse/ARROW-18198). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
貢獻指南
研究方向
從 ArrowReader.loadRecordBatch 開始,沿著 VectorLoader、VectorSchemaRoot.setRowCount 和 BaseFixedWidthVector.reAlloc 追蹤呼叫堆疊。使用提供的 pandas 壓縮 Feather 範例重現問題,將其與未壓縮檔案進行比較,並確認載入壓縮 dataframe 時不會出現 IndexOutOfBoundsException 且能夠完成。
由索引模型根據 Issue 內容生成。
評估
- 技術堆疊
- java, pandas
- 領域
- data
- Issue 類型
- 缺陷
- 難度
- 4/5
- 預估耗時
- 3-5 天
- 活躍度
- 停滯
- 描述清晰度
- 基本清楚
- 新手友好度
- 35/100