ValueError: need at least one array to concatenate When Returned Dataset is Emtpy
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 55/100
Research direction
Start in databricks/sql/client.py at fetchall, fetchall_arrow, and _convert_arrow_table, following the failing conversion path shown in the traceback. Reproduce the SELECT ... WHERE 1=0 case and verify that fetchall returns an empty list instead of raising ValueError.
Written by the indexing model from the issue text.
Description
I notice when a query returns empty dataset, the fetchall function fails with ValueError: need at least one array to concatenate. The expected behavior is to return an empty list.
python 3.8
databricks-sql-connector==3.7.1
pyarrow==17.0.0
sql_query = f"""
SELECT Column FROM table
WHERE 1=0"""
host = os.getenv("DATABRICKS_HOST")
http_path = os.getenv("DATABRICKS_HTTP_PATH")
connection = sql.connect(
server_hostname=host,
http_path=http_path)
with conn.cursor() as cursor:
cursor.execute(sql_query)
rows = cursor.fetchall()
File "c:\Users\acai\.venv\lib\site-packages\databricks\sql\client.py", line 1066, in fetchall
return self.active_result_set.fetchall()
File "c:\Users\acai\.venv\lib\site-packages\databricks\sql\client.py", line 1431, in fetchall
return self._convert_arrow_table(self.fetchall_arrow())
File "c:\Users\acai\.venv\lib\site-packages\databricks\sql\client.py", line 1302, in _convert_arrow_table
df = table_renamed.to_pandas(
File "pyarrow\\array.pxi", line 885, in pyarrow.lib._PandasConvertible.to_pandas
File "pyarrow\\table.pxi", line 5002, in pyarrow.lib.Table._to_pandas
File "c:\Users\acai\.venv\lib\site-packages\pyarrow\pandas_compat.py", line 800, in table_to_dataframe
blocks = [
File "c:\Users\acai\O.venv\lib\site-packages\pyarrow\pandas_compat.py", line 801, in <listcomp>
_reconstruct_block(item, column_names, ext_columns_dtypes)
File "c:\Users\acai\.venv\lib\site-packages\pyarrow\pandas_compat.py", line 743, in _reconstruct_block
arr = pandas_dtype.__from_arrow__(arr)
File "c:\Users\acai\.venv\lib\site-packages\pandas\core\arrays\integer.py", line 121, in __from_arrow__
return IntegerArray._concat_same_type(results)
File "c:\Users\acai\\.venv\lib\site-packages\pandas\core\arrays\masked.py", line 271, in _concat_same_type
data = np.concatenate([x._data for x in to_concat])
File "<__array_function__ internals>", line 200, in concatenate
ValueError: need at least one array to concatenate
- Dominant language
- Python
- Stars
- 233
- Forks
- 152
- Avg merge
- 21h 5m
- Merged PRs (30d)
- 10
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from databricks/databricks-sql-python
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
All issues in databricks/databricks-sql-python
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
-
🐛 Bug 🔔 Pending processing
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
jumpserver/jumpserver#17584 ·