apache / apache/arrow-java

[Java] The ArrowIPC Java library doesn't respect buffer offsets set in the flatbuffer metadata

未关闭
#221 5 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
Type: bug
主要语言
Java
星标
94
派生
152
平均合并
3 天 16 小时
30 天内合并 PR
11

描述

### Describe the bug, including details regarding any error messages, version, and platform.

The `Buffer Alignment and Padding` [section](https://arrow.apache.org/docs/format/Columnar.html#buffer-alignment-and-padding) of the Arrow Columnar Format Specification allows padding to be either 8-bytes or 64-bytes with 64-bytes being preferred for SIMD performance reasons.

I have a custom implementation of the Arrow Columnar Format (serialization and deserialization only as we operate on an entirely different columnar format) where I pad all of my data buffers to 64-bytes when serializing. This works fine with the C++, Python and Rust libraries (and I think with the Go library as well) as they all use the `offset` value from the flatbuffer `Buffer` message when calculating where to read the next body buffer from...

The Java library however, seems to assume that the padding will always be 8-bytes and just pads the size of the `n-1`th buffer to 8-bytes when calculating the offset for the `n`th buffer. This assumption obviously breaks when the padding size is anything other than 8-bytes...

Would it be acceptable to bring the Java library in-line with the remaining implementations, i.e., have it respect the offset from the flatbuffer metadata?

### Component(s)

Java

贡献指南

打开贡献指南

调研方向

首先定位 Java 反序列化逻辑,该逻辑根据 flatbuffer Buffer 元数据计算连续的 body-buffer 位置。将其偏移量计算与 Arrow Columnar Format 规范进行比较,然后使用带有 64 字节填充的序列化输入验证 fix,并确认所有 Buffer 都从其元数据偏移量处读取。

由索引模型根据 Issue 内容生成。

评估

技术栈
java
领域
data-engineering
Issue 类型
缺陷
难度
4/5
预计耗时
3-5 天
活跃度
停滞
描述清晰度
基本清楚
新手友好度
45/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。