apache / apache/arrow-java

[Flight][Java][C++] Data read through Flight is having endianness issue on s390x

Aperta
#352 8 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Type: bug
Lingua principale
Java
Stelle
94
Fork
152
Merge medio
3g 16h
PR unite (30g)
11

Descrizione

Am facing an endianness issue on s390x(big endian) when converting the data read through flight to pandas data frame.

(1) table.validate() fails with error
```Java

Traceback (most recent call last):
  File "/tmp/2.py", line 51, in
    table.validate()
  File "pyarrow/table.pxi", line 1232, in pyarrow.lib.Table.validate
  File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Column 1: In chunk 0: Invalid: Negative offsets in binary array
```

(2) table.to_pandas() gives a segmentation fault
____________
Here is a sample code that I am using:
```python

from pyarrow import flight
import os
import json

flight_endpoint = os.environ.get("flight_server_url", "grpc+tls://...local:443")
print(flight_endpoint)

#
class TokenClientAuthHandler(flight.ClientAuthHandler):
    """An example implementation of authentication via handshake.
       With the default constructor, the user token is read from the environment: TokenClientAuthHandler().
       You can also pass a user token as parameter to the constructor, TokenClientAuthHandler(yourtoken).
    """
    def \_\_init\_\_(self, token: str = None):
        super().\_\_init\__()
        if( token != None):
            strToken = strToken = 'Bearer {}'.format(token)
        else:
            strToken = 'Bearer {}'.format(os.environ.get("some_auth_token"))
        self.token = strToken.encode('utf-8')
        #print(self.token)

    def authenticate(self, outgoing, incoming):
        outgoing.write(self.token)
        self.token = incoming.read()

    def get_token(self):
        return self.token
    
readClient = flight.FlightClient(flight_endpoint)
readClient.authenticate(TokenClientAuthHandler())

cmd = json.dumps(\{...})

descriptor = flight.FlightDescriptor.for_command(cmd)
flightInfo = readClient.get_flight_info(descriptor)

reader = readClient.do_get(flightInfo.endpoints[0].ticket)
table = reader.read_all()

print(table)
print(table.num_columns)
print(table.num_rows)
table.validate()
table.to_pandas()
```

**Environment**: Linux s390x (big endian)
**Reporter**: [Ravi Gummadi](https://issues.apache.org/jira/browse/ARROW-15645)

**Note**: *This issue was originally created as [ARROW-15645](https://issues.apache.org/jira/browse/ARROW-15645). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Use the provided Python reproduction on Linux s390x, starting at FlightClient.do_get(), reader.read_all(), and the pyarrow table validation path in pyarrow/table.pxi and error.pxi. Trace how the Flight data is interpreted on big-endian systems and verify that table.validate() and table.to_pandas() complete without invalid offsets or a segmentation fault.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
cpp, java, python
Ambito
data-engineering, distributed-systems
Tipo di issue
Bug
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Ferma
Chiarezza
Da chiarire
Idoneità per principianti
30/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.