[Java] The ArrowIPC Java library doesn't respect buffer offsets set in the flatbuffer metadata
- Langage dominant
- Java
- Étoiles
- 94
- Forks
- 152
- Merge moyen
- 3 j 16 h
- PR mergées (30 j)
- 11
Description
### Describe the bug, including details regarding any error messages, version, and platform.
The `Buffer Alignment and Padding` [section](https://arrow.apache.org/docs/format/Columnar.html#buffer-alignment-and-padding) of the Arrow Columnar Format Specification allows padding to be either 8-bytes or 64-bytes with 64-bytes being preferred for SIMD performance reasons.
I have a custom implementation of the Arrow Columnar Format (serialization and deserialization only as we operate on an entirely different columnar format) where I pad all of my data buffers to 64-bytes when serializing. This works fine with the C++, Python and Rust libraries (and I think with the Go library as well) as they all use the `offset` value from the flatbuffer `Buffer` message when calculating where to read the next body buffer from...
The Java library however, seems to assume that the padding will always be 8-bytes and just pads the size of the `n-1`th buffer to 8-bytes when calculating the offset for the `n`th buffer. This assumption obviously breaks when the padding size is anything other than 8-bytes...
Would it be acceptable to bring the Java library in-line with the remaining implementations, i.e., have it respect the offset from the flatbuffer metadata?
### Component(s)
Java
Guide de contribution
Ouvrir le guide de contribution
Piste de recherche
Commencez par localiser la logique de désérialisation Java qui calcule les positions successives dans le body-buffer à partir des métadonnées de Buffer de flatbuffer. Comparez son calcul d’offset avec la spécification d’Arrow Columnar Format, puis vérifiez le fix à l’aide d’une entrée sérialisée avec un padding de 64 octets et confirmez que tous les Buffer sont lus depuis leurs offsets de métadonnées.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Évaluation
- Stack technique
- java
- Domaine
- data-engineering
- Type d'issue
- Bug
- Difficulté
- 4/5
- Temps estimé
- 3-5 jours
- Activité
- À l'abandon
- Clarté
- Plutôt claire
- Accessibilité débutants
- 45/100