apache / apache/arrow-java

[Java] The ArrowIPC Java library doesn't respect buffer offsets set in the flatbuffer metadata

Abierto
#221 5 comentarios 0 reacciones 0 asignados Ver en GitHub
Type: bug
Lenguaje dominante
Java
Estrellas
94
Forks
152
Merge medio
3 d 16 h
PR fusionados (30 d)
11

Descripción

### Describe the bug, including details regarding any error messages, version, and platform.

The `Buffer Alignment and Padding` [section](https://arrow.apache.org/docs/format/Columnar.html#buffer-alignment-and-padding) of the Arrow Columnar Format Specification allows padding to be either 8-bytes or 64-bytes with 64-bytes being preferred for SIMD performance reasons.

I have a custom implementation of the Arrow Columnar Format (serialization and deserialization only as we operate on an entirely different columnar format) where I pad all of my data buffers to 64-bytes when serializing. This works fine with the C++, Python and Rust libraries (and I think with the Go library as well) as they all use the `offset` value from the flatbuffer `Buffer` message when calculating where to read the next body buffer from...

The Java library however, seems to assume that the padding will always be 8-bytes and just pads the size of the `n-1`th buffer to 8-bytes when calculating the offset for the `n`th buffer. This assumption obviously breaks when the padding size is anything other than 8-bytes...

Would it be acceptable to bring the Java library in-line with the remaining implementations, i.e., have it respect the offset from the flatbuffer metadata?

### Component(s)

Java

Guía de contribución

Abrir la guía de contribución

Línea de trabajo

Comienza localizando la lógica de deserialización de Java que calcula las posiciones sucesivas del body-buffer a partir de los metadatos de Buffer de flatbuffer. Compara su cálculo de offsets con la especificación de Arrow Columnar Format, luego verifica el fix usando una entrada serializada con padding de 64 bytes y confirma que todos los Buffer se leen desde sus offsets de metadatos.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
java
Área
data-engineering
Tipo de issue
Error
Dificultad
4/5
Tiempo estimado
3-5 días
Estado de actividad
Estancado
Claridad
Bastante claro
Aptitud para principiantes
45/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.