[Java] The ArrowIPC Java library doesn't respect buffer offsets set in the flatbuffer metadata
- Dominant language
- Java
- Stars
- 94
- Forks
- 152
- Avg merge
- 3d 16h
- Merged PRs (30d)
- 11
Description
### Describe the bug, including details regarding any error messages, version, and platform.
The `Buffer Alignment and Padding` [section](https://arrow.apache.org/docs/format/Columnar.html#buffer-alignment-and-padding) of the Arrow Columnar Format Specification allows padding to be either 8-bytes or 64-bytes with 64-bytes being preferred for SIMD performance reasons.
I have a custom implementation of the Arrow Columnar Format (serialization and deserialization only as we operate on an entirely different columnar format) where I pad all of my data buffers to 64-bytes when serializing. This works fine with the C++, Python and Rust libraries (and I think with the Go library as well) as they all use the `offset` value from the flatbuffer `Buffer` message when calculating where to read the next body buffer from...
The Java library however, seems to assume that the padding will always be 8-bytes and just pads the size of the `n-1`th buffer to 8-bytes when calculating the offset for the `n`th buffer. This assumption obviously breaks when the padding size is anything other than 8-bytes...
Would it be acceptable to bring the Java library in-line with the remaining implementations, i.e., have it respect the offset from the flatbuffer metadata?
### Component(s)
Java
Contributor guide
Research direction
Start by locating the Java deserialization logic that calculates successive body-buffer positions from flatbuffer Buffer metadata. Compare its offset calculation with the Arrow Columnar Format specification, then verify the fix using a serialized input with 64-byte padding and confirm that all buffers are read from their metadata offsets.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100