[Java] The ArrowIPC Java library doesn't respect buffer offsets set in the flatbuffer metadata
- Vorherrschende Sprache
- Java
- Sterne
- 94
- Forks
- 152
- Ø Merge
- 3 T. 16 Std.
- Gemergte PRs (30 T.)
- 11
Beschreibung
### Describe the bug, including details regarding any error messages, version, and platform.
The `Buffer Alignment and Padding` [section](https://arrow.apache.org/docs/format/Columnar.html#buffer-alignment-and-padding) of the Arrow Columnar Format Specification allows padding to be either 8-bytes or 64-bytes with 64-bytes being preferred for SIMD performance reasons.
I have a custom implementation of the Arrow Columnar Format (serialization and deserialization only as we operate on an entirely different columnar format) where I pad all of my data buffers to 64-bytes when serializing. This works fine with the C++, Python and Rust libraries (and I think with the Go library as well) as they all use the `offset` value from the flatbuffer `Buffer` message when calculating where to read the next body buffer from...
The Java library however, seems to assume that the padding will always be 8-bytes and just pads the size of the `n-1`th buffer to 8-bytes when calculating the offset for the `n`th buffer. This assumption obviously breaks when the padding size is anything other than 8-bytes...
Would it be acceptable to bring the Java library in-line with the remaining implementations, i.e., have it respect the offset from the flatbuffer metadata?
### Component(s)
Java
Beitragsleitfaden
Rechercherichtung
Beginne damit, die Java-Deserialisierungslogik zu finden, die auf Grundlage der flatbuffer Buffer-Metadaten aufeinanderfolgende Positionen im Body-Buffer berechnet. Vergleiche ihre Offset-Berechnung mit der Spezifikation des Arrow Columnar Format, überprüfe anschließend den Fix anhand einer serialisierten Eingabe mit 64-Byte-Padding und bestätige, dass alle Buffer von ihren Metadaten-Offsets gelesen werden.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Bewertung
- Tech-Stack
- java
- Bereich
- data-engineering
- Issue-Typ
- Bug
- Schwierigkeit
- 4/5
- Geschätzter Aufwand
- 3-5 Tage
- Aktivitätsstatus
- Veraltet
- Klarheit
- Größtenteils klar
- Anfängerfreundlichkeit
- 45/100