apache / apache/arrow-java

[Flight][Java][C++] Data read through Flight is having endianness issue on s390x

Abierto
#352 8 comentarios 0 reacciones 0 asignados Ver en GitHub
Type: bug
Lenguaje dominante
Java
Estrellas
94
Forks
152
Merge medio
3 d 16 h
PR fusionados (30 d)
11

Descripción

Am facing an endianness issue on s390x(big endian) when converting the data read through flight to pandas data frame.

(1) table.validate() fails with error
```Java

Traceback (most recent call last):
  File "/tmp/2.py", line 51, in
    table.validate()
  File "pyarrow/table.pxi", line 1232, in pyarrow.lib.Table.validate
  File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Column 1: In chunk 0: Invalid: Negative offsets in binary array
```

(2) table.to_pandas() gives a segmentation fault
____________
Here is a sample code that I am using:
```python

from pyarrow import flight
import os
import json

flight_endpoint = os.environ.get("flight_server_url", "grpc+tls://...local:443")
print(flight_endpoint)

#
class TokenClientAuthHandler(flight.ClientAuthHandler):
    """An example implementation of authentication via handshake.
       With the default constructor, the user token is read from the environment: TokenClientAuthHandler().
       You can also pass a user token as parameter to the constructor, TokenClientAuthHandler(yourtoken).
    """
    def \_\_init\_\_(self, token: str = None):
        super().\_\_init\__()
        if( token != None):
            strToken = strToken = 'Bearer {}'.format(token)
        else:
            strToken = 'Bearer {}'.format(os.environ.get("some_auth_token"))
        self.token = strToken.encode('utf-8')
        #print(self.token)

    def authenticate(self, outgoing, incoming):
        outgoing.write(self.token)
        self.token = incoming.read()

    def get_token(self):
        return self.token
    
readClient = flight.FlightClient(flight_endpoint)
readClient.authenticate(TokenClientAuthHandler())

cmd = json.dumps(\{...})

descriptor = flight.FlightDescriptor.for_command(cmd)
flightInfo = readClient.get_flight_info(descriptor)

reader = readClient.do_get(flightInfo.endpoints[0].ticket)
table = reader.read_all()

print(table)
print(table.num_columns)
print(table.num_rows)
table.validate()
table.to_pandas()
```

**Environment**: Linux s390x (big endian)
**Reporter**: [Ravi Gummadi](https://issues.apache.org/jira/browse/ARROW-15645)

**Note**: *This issue was originally created as [ARROW-15645](https://issues.apache.org/jira/browse/ARROW-15645). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Guía de contribución

Abrir la guía de contribución

Línea de trabajo

Use the provided Python reproduction on Linux s390x, starting at FlightClient.do_get(), reader.read_all(), and the pyarrow table validation path in pyarrow/table.pxi and error.pxi. Trace how the Flight data is interpreted on big-endian systems and verify that table.validate() and table.to_pandas() complete without invalid offsets or a segmentation fault.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
cpp, java, python
Área
data-engineering, distributed-systems
Tipo de issue
Error
Dificultad
4/5
Tiempo estimado
3-5 días
Estado de actividad
Estancado
Claridad
Necesita aclaración
Aptitud para principiantes
30/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.