ValueError: need at least one array to concatenate When Returned Dataset is Emtpy

Open
#492 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
55/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Quiet
Tech stack
python, sql
Domain
api, database

Research direction

Start in databricks/sql/client.py at fetchall, fetchall_arrow, and _convert_arrow_table, following the failing conversion path shown in the traceback. Reproduce the SELECT ... WHERE 1=0 case and verify that fetchall returns an empty list instead of raising ValueError.

Written by the indexing model from the issue text.

Description

engineer-bot

I notice when a query returns empty dataset, the fetchall function fails with ValueError: need at least one array to concatenate. The expected behavior is to return an empty list.

python 3.8
databricks-sql-connector==3.7.1
pyarrow==17.0.0

   sql_query = f"""
                SELECT Column FROM table
                WHERE 1=0"""

host = os.getenv("DATABRICKS_HOST")
http_path = os.getenv("DATABRICKS_HTTP_PATH")

connection = sql.connect(
  server_hostname=host,
  http_path=http_path)
    with conn.cursor() as cursor:
        cursor.execute(sql_query)
        rows = cursor.fetchall()
  File "c:\Users\acai\.venv\lib\site-packages\databricks\sql\client.py", line 1066, in fetchall
    return self.active_result_set.fetchall()
  File "c:\Users\acai\.venv\lib\site-packages\databricks\sql\client.py", line 1431, in fetchall
    return self._convert_arrow_table(self.fetchall_arrow())
  File "c:\Users\acai\.venv\lib\site-packages\databricks\sql\client.py", line 1302, in _convert_arrow_table
    df = table_renamed.to_pandas(
  File "pyarrow\\array.pxi", line 885, in pyarrow.lib._PandasConvertible.to_pandas
  File "pyarrow\\table.pxi", line 5002, in pyarrow.lib.Table._to_pandas
  File "c:\Users\acai\.venv\lib\site-packages\pyarrow\pandas_compat.py", line 800, in table_to_dataframe
    blocks = [
  File "c:\Users\acai\O.venv\lib\site-packages\pyarrow\pandas_compat.py", line 801, in <listcomp>
    _reconstruct_block(item, column_names, ext_columns_dtypes)
  File "c:\Users\acai\.venv\lib\site-packages\pyarrow\pandas_compat.py", line 743, in _reconstruct_block
    arr = pandas_dtype.__from_arrow__(arr)
  File "c:\Users\acai\.venv\lib\site-packages\pandas\core\arrays\integer.py", line 121, in __from_arrow__
    return IntegerArray._concat_same_type(results)
  File "c:\Users\acai\\.venv\lib\site-packages\pandas\core\arrays\masked.py", line 271, in _concat_same_type
    data = np.concatenate([x._data for x in to_concat])
  File "<__array_function__ internals>", line 200, in concatenate
ValueError: need at least one array to concatenate

Dominant language
Python
Stars
233
Forks
152
Avg merge
21h 5m
Merged PRs (30d)
10

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from databricks/databricks-sql-python

All issues in databricks/databricks-sql-python

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.