Arrow vector in Java(Scala) allocate byteBuffer error while read the bytes from Python pyarrow
- 主要語言
- Java
- 星號
- 94
- 分支
- 152
- 平均合併
- 3 天 16 小時
- 30 天內合併 PR
- 11
描述
I am using scala arrow 1.0.1 and pyarrow 1.0.1
Following error occurs when scala decode the byte that encoded from python.
tried to downgrade pyarrow to 0.17.0, 0.14.1, error still exists.
This is a loop function to decode the same data repeatedly. The error occurs sometimes, after an error occurs, following code could still work correctly.
```
Error stack trace java.nio.ByteBuffer.allocate(ByteBuffer.java:334)
com.intel.analytics.zoo.shaded.arrow.vector.ipc.message.MessageSerializer.readMessage(MessageSerializer.java:692)
com.intel.analytics.zoo.shaded.arrow.vector.ipc.message.MessageChannelReader.readNext(MessageChannelReader.java:57)
com.intel.analytics.zoo.shaded.arrow.vector.ipc.ArrowStreamReader.readSchema(ArrowStreamReader.java:164)
com.intel.analytics.zoo.shaded.arrow.vector.ipc.ArrowReader.initialize(ArrowReader.java:170)
com.intel.analytics.zoo.shaded.arrow.vector.ipc.ArrowReader.ensureInitialized(ArrowReader.java:161)
com.intel.analytics.zoo.shaded.arrow.vector.ipc.ArrowReader.getVectorSchemaRoot(ArrowReader.java:63)
```
How to fix it?
**Reporter**: [Litchy Soong](https://issues.apache.org/jira/browse/ARROW-10341)
**Note**: *This issue was originally created as [ARROW-10341](https://issues.apache.org/jira/browse/ARROW-10341). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
貢獻指南
研究方向
從 com.intel.analytics.zoo.shaded.arrow.vector.ipc.message.MessageSerializer.readMessage 開始,沿著 MessageChannelReader.readNext 和 ArrowStreamReader.readSchema 追蹤呼叫堆疊。重複執行 Scala 對由 Python pyarrow 1.0.1 編碼之位元組的讀取,著重於間歇性的 ByteBuffer 配置失敗;當重複解碼不再間歇性失敗時即完成。
由索引模型根據 Issue 內容生成。
評估
- 技術堆疊
- java, python, scala
- 領域
- data-engineering
- Issue 類型
- 缺陷
- 難度
- 4/5
- 預估耗時
- 3-5 天
- 活躍度
- 停滯
- 描述清晰度
- 需要釐清
- 新手友好度
- 25/100