apache / apache/arrow-java

[Flight][Java][C++] Data read through Flight is having endianness issue on s390x

Aberta
#352 8 comentários 0 reações 0 responsáveis Ver no GitHub
Type: bug
Linguagem predominante
Java
Estrelas
94
Forks
152
Merge médio
3d 16h
PRs com merge (30d)
11

Descrição

Am facing an endianness issue on s390x(big endian) when converting the data read through flight to pandas data frame.

(1) table.validate() fails with error
```Java

Traceback (most recent call last):
  File "/tmp/2.py", line 51, in
    table.validate()
  File "pyarrow/table.pxi", line 1232, in pyarrow.lib.Table.validate
  File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Column 1: In chunk 0: Invalid: Negative offsets in binary array
```

(2) table.to_pandas() gives a segmentation fault
____________
Here is a sample code that I am using:
```python

from pyarrow import flight
import os
import json

flight_endpoint = os.environ.get("flight_server_url", "grpc+tls://...local:443")
print(flight_endpoint)

#
class TokenClientAuthHandler(flight.ClientAuthHandler):
    """An example implementation of authentication via handshake.
       With the default constructor, the user token is read from the environment: TokenClientAuthHandler().
       You can also pass a user token as parameter to the constructor, TokenClientAuthHandler(yourtoken).
    """
    def \_\_init\_\_(self, token: str = None):
        super().\_\_init\__()
        if( token != None):
            strToken = strToken = 'Bearer {}'.format(token)
        else:
            strToken = 'Bearer {}'.format(os.environ.get("some_auth_token"))
        self.token = strToken.encode('utf-8')
        #print(self.token)

    def authenticate(self, outgoing, incoming):
        outgoing.write(self.token)
        self.token = incoming.read()

    def get_token(self):
        return self.token
    
readClient = flight.FlightClient(flight_endpoint)
readClient.authenticate(TokenClientAuthHandler())

cmd = json.dumps(\{...})

descriptor = flight.FlightDescriptor.for_command(cmd)
flightInfo = readClient.get_flight_info(descriptor)

reader = readClient.do_get(flightInfo.endpoints[0].ticket)
table = reader.read_all()

print(table)
print(table.num_columns)
print(table.num_rows)
table.validate()
table.to_pandas()
```

**Environment**: Linux s390x (big endian)
**Reporter**: [Ravi Gummadi](https://issues.apache.org/jira/browse/ARROW-15645)

**Note**: *This issue was originally created as [ARROW-15645](https://issues.apache.org/jira/browse/ARROW-15645). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Guia de contribuição

Abrir o guia de contribuição

Direção de pesquisa

Use the provided Python reproduction on Linux s390x, starting at FlightClient.do_get(), reader.read_all(), and the pyarrow table validation path in pyarrow/table.pxi and error.pxi. Trace how the Flight data is interpreted on big-endian systems and verify that table.validate() and table.to_pandas() complete without invalid offsets or a segmentation fault.

Escrita pelo modelo de indexação a partir do texto da issue.

Avaliação

Stack de tecnologia
cpp, java, python
Domínio
data-engineering, distributed-systems
Tipo de issue
Bug
Dificuldade
4/5
Tempo estimado
3-5 dias
Status de atividade
Estagnada
Clareza
Precisa de esclarecimento
Facilidade para iniciantes
30/100

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.