apache / apache/arrow-java

[Flight][Java][C++] Data read through Flight is having endianness issue on s390x

オープン
#352 コメント 8 件 リアクション 0 件 担当者 0 名 GitHub で見る
Type: bug
主要言語
Java
スター
94
フォーク
152
平均マージ
3日 16時間
マージ済み PR(30日)
11

説明

Am facing an endianness issue on s390x(big endian) when converting the data read through flight to pandas data frame.

(1) table.validate() fails with error
```Java

Traceback (most recent call last):
  File "/tmp/2.py", line 51, in
    table.validate()
  File "pyarrow/table.pxi", line 1232, in pyarrow.lib.Table.validate
  File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Column 1: In chunk 0: Invalid: Negative offsets in binary array
```

(2) table.to_pandas() gives a segmentation fault
____________
Here is a sample code that I am using:
```python

from pyarrow import flight
import os
import json

flight_endpoint = os.environ.get("flight_server_url", "grpc+tls://...local:443")
print(flight_endpoint)

#
class TokenClientAuthHandler(flight.ClientAuthHandler):
    """An example implementation of authentication via handshake.
       With the default constructor, the user token is read from the environment: TokenClientAuthHandler().
       You can also pass a user token as parameter to the constructor, TokenClientAuthHandler(yourtoken).
    """
    def \_\_init\_\_(self, token: str = None):
        super().\_\_init\__()
        if( token != None):
            strToken = strToken = 'Bearer {}'.format(token)
        else:
            strToken = 'Bearer {}'.format(os.environ.get("some_auth_token"))
        self.token = strToken.encode('utf-8')
        #print(self.token)

    def authenticate(self, outgoing, incoming):
        outgoing.write(self.token)
        self.token = incoming.read()

    def get_token(self):
        return self.token
    
readClient = flight.FlightClient(flight_endpoint)
readClient.authenticate(TokenClientAuthHandler())

cmd = json.dumps(\{...})

descriptor = flight.FlightDescriptor.for_command(cmd)
flightInfo = readClient.get_flight_info(descriptor)

reader = readClient.do_get(flightInfo.endpoints[0].ticket)
table = reader.read_all()

print(table)
print(table.num_columns)
print(table.num_rows)
table.validate()
table.to_pandas()
```

**Environment**: Linux s390x (big endian)
**Reporter**: [Ravi Gummadi](https://issues.apache.org/jira/browse/ARROW-15645)

**Note**: *This issue was originally created as [ARROW-15645](https://issues.apache.org/jira/browse/ARROW-15645). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

コントリビューションガイド

コントリビューションガイドを開く

調査の方向性

Use the provided Python reproduction on Linux s390x, starting at FlightClient.do_get(), reader.read_all(), and the pyarrow table validation path in pyarrow/table.pxi and error.pxi. Trace how the Flight data is interpreted on big-endian systems and verify that table.validate() and table.to_pandas() complete without invalid offsets or a segmentation fault.

索引モデルが issue の本文から書いたものです。

評価

技術スタック
cpp, java, python
領域
data-engineering, distributed-systems
issue の種類
バグ
難易度
4/5
見積もり時間
3〜5日
活発さ
停滞
明瞭さ
説明が足りない
初心者へのやさしさ
30/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。