apache / apache/arrow

Unicode character issue with pyarrow

Open
#32,384 0 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

When running code using databricks SQL connector for Python, it hit a unicode character issue in pyarrow library. The customer has to put a workaround in the client code, something like
"SELECT decode(string(unbase64(value)), 'utf8')"
 
Exception in the main script No data fetched using SQL-statement: SELECT \* FROM parquet.`abfss://XXXX@XXXX.xxx.net/structXXXXXXX`. Exception: Unknown error: Wrapping TP H�kan  Sweater failed Traceback (most recent call last):  
File "/home/xxxx/yy/allo/yy/db/sql_reader.py", line 53, in query     rows = cursor.fetchmany(self.MAX_ROWS)  
File "/home/xxxx/yy/.venv/lib/python3.10/site-packages/databricks/sql/client.py", line 401, in fetchmany     return self.active_result_set.fetchmany(size)  
File "/home/xxxx/yy/.venv/lib/python3.10/site-packages/databricks/sql/client.py", line 630, in fetchmany     return self._convert_arrow_table(self.fetchmany_arrow(size))  
File "/home/xxxx/yy/.venv/lib/python3.10/site-packages/databricks/sql/client.py", line 563, in _convert_arrow_table     df = table_renamed.to_pandas(  
File "pyarrow/array.pxi", line 822, in pyarrow.lib._PandasConvertible.to_pandas  
File "pyarrow/table.pxi", line 3889, in pyarrow.lib.Table._to_pandas  
File "/home/xxxx/yy/.venv/lib/python3.10/site-packages/pyarrow/pandas_compat.py", line 803, in table_to_blockmanager     blocks = _table_to_blocks(options, table, categories, ext_columns_dtypes)  
File "/home/xxxx/yy/.venv/lib/python3.10/site-packages/pyarrow/pandas_compat.py", line 1155, in _table_to_blocks     return [_reconstruct_block(item, columns, extension_columns)  
File "/home/xxxx/yy/.venv/lib/python3.10/site-packages/pyarrow/pandas_compat.py", line 1155, in     return [_reconstruct_block(item, columns, extension_columns)  
File "/home/xxxx/yy/.venv/lib/python3.10/site-packages/pyarrow/pandas_compat.py", line 763, in _reconstruct_block     pd_ext_arr = pandas_dtype.__from_arrow__(arr)  
File "/home/xxxx/yy/.venv/lib/python3.10/site-packages/pandas/core/arrays/string_.py", line 217, in __from_arrow__     str_arr = StringArray._from_sequence(np.array(arr))  
File "pyarrow/array.pxi", line 1395, in pyarrow.lib.Array.__array__  
File "pyarrow/array.pxi", line 1441, in pyarrow.lib.Array.to_numpy  
File "pyarrow/error.pxi", line 138, in pyarrow.lib.check_status pyarrow.lib.ArrowException: Unknown error: Wrapping TP H�kan  Sweater failed During handling of the above exception, another exception occurred: Traceback (most recent call last):  

**Reporter**: [Yunbo Deng](https://issues.apache.org/jira/browse/ARROW-17077)

**Note**: *This issue was originally created as [ARROW-17077](https://issues.apache.org/jira/browse/ARROW-17077). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the Unicode failure described in the Databricks SQL connector traceback, following the pyarrow conversion path through pandas_compat.py and pandas/core/arrays/string_.py. Confirm that the reported value converts without the ArrowException and add or update coverage for the failing Unicode case.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.