apache / apache/datafusion

ArrowInvalid Error when converting DataFrame to pandas

Open
#12,354 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

I am testing the example from [sql-to-pandas.py](https://github.com/apache/datafusion-python/blob/main/examples/sql-to-pandas.py) and I got en error ArrowInvalid with different schema on the line: pandas_df = df.to_pandas().

ArrowInvalid: Schema at index 0 was different:
passenger_count: int64
trip_count: int32 not null
vs
passenger_count: int64
trip_count: int32

Is there any way how to define trip_acount as not nullable when running select or how to make pandas to accept such a DataFrame.

I am testing with conda env with python 3.11.9 on windows. Pip packages I have listed bellow.

```
---------------------------------------------------------------------------
ArrowInvalid Traceback (most recent call last)
Cell In[11], [line 19](vscode-notebook-cell:?execution_count=11&line=19)
[17](vscode-notebook-cell:?execution_count=11&line=17) print(df.schema())
[18](vscode-notebook-cell:?execution_count=11&line=18) # convert to Pandas
---> [19](vscode-notebook-cell:?execution_count=11&line=19) pandas_df = df.to_pandas()
[21](vscode-notebook-cell:?execution_count=11&line=21) # create a chart
[22](vscode-notebook-cell:?execution_count=11&line=22) fig = pandas_df.plot(kind="bar", title="Trip Count by Number of Passengers").get_figure()

File c:\Users\LDolezal\Miniconda3\envs\pyarrow\Lib\site-packages\datafusion\dataframe.py:472, in DataFrame.to_pandas(self)
[466](file:///C:/Users/LDolezal/Miniconda3/envs/pyarrow/Lib/site-packages/datafusion/dataframe.py:466) def to_pandas(self) -> pd.DataFrame:
[467](file:///C:/Users/LDolezal/Miniconda3/envs/pyarrow/Lib/site-packages/datafusion/dataframe.py:467) """Execute the :py:class:`DataFrame` and convert it into a Pandas DataFrame.
[468](file:///C:/Users/LDolezal/Miniconda3/envs/pyarrow/Lib/site-packages/datafusion/dataframe.py:468)
[469](file:///C:/Users/LDolezal/Miniconda3/envs/pyarrow/Lib/site-packages/datafusion/dataframe.py:469) Returns:
[470](file:///C:/Users/LDolezal/Miniconda3/envs/pyarrow/Lib/site-packages/datafusion/dataframe.py:470) Pandas DataFrame.
[471](file:///C:/Users/LDolezal/Miniconda3/envs/pyarrow/Lib/site-packages/datafusion/dataframe.py:471) """
--> [472](file:///C:/Users/LDolezal/Miniconda3/envs/pyarrow/Lib/site-packages/datafusion/dataframe.py:472) return self.df.to_pandas()

File c:\Users\LDolezal\Miniconda3\envs\pyarrow\Lib\site-packages\pyarrow\table.pxi:4865, in pyarrow.lib.Table.from_batches()

File c:\Users\LDolezal\Miniconda3\envs\pyarrow\Lib\site-packages\pyarrow\error.pxi:155, in pyarrow.lib.pyarrow_internal_check_status()

File c:\Users\LDolezal\Miniconda3\envs\pyarrow\Lib\site-packages\pyarrow\error.pxi:92, in pyarrow.lib.check_status()

ArrowInvalid: Schema at index 0 was different:
passenger_count: int64
trip_count: int32 not null
vs
passenger_count: int64
trip_count: int32
```

Data source for the data is here:
https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2024-01.parquet

### To Reproduce

```
from datafusion import SessionContext

ctx = SessionContext()

# Register table with context
ctx.register_parquet('taxi', 'datasets/yellow_tripdata_2024-01.parquet')

# Execute SQL
df = ctx.sql("select passenger_count, cast(count(*) as int) as trip_count "
"from taxi "
"where passenger_count is not null "
"group by passenger_count "
"order by passenger_count")
# convert to Pandas

df.show()
print(df.schema())
# convert to Pandas
pandas_df = df.to_pandas()

# create a chart
fig = pandas_df.plot(kind="bar", title="Trip Count by Number of Passengers").get_figure()
fig.savefig('chart.png')

```

### Expected behavior

_No response_

### Additional context

Package Version
----------------- -----------
asttokens 2.0.5
colorama 0.4.6
comm 0.2.1
datafusion 40.1.0
debugpy 1.6.7
decorator 5.1.1
executing 0.8.3
ipykernel 6.28.0
ipython 8.25.0
jedi 0.19.1
jupyter_client 8.6.0
jupyter_core 5.7.2
matplotlib-inline 0.1.6
nest-asyncio 1.6.0
numpy 2.1.1
packaging 24.1
pandas 2.2.2
parso 0.8.3
pip 24.2
platformdirs 3.10.0
prompt-toolkit 3.0.43
psutil 5.9.0
pure-eval 0.2.2
pyarrow 17.0.0
Pygments 2.15.1
python-dateutil 2.9.0.post0
pytz 2024.1
pywin32 305.1
pyzmq 25.1.2
setuptools 72.1.0
six 1.16.0
stack-data 0.2.0
tornado 6.4.1
traitlets 5.14.3
typing_extensions 4.11.0
tzdata 2024.1
wcwidth 0.2.5
wheel 0.43.0

Contributor guide

Open the contributing guide

Research direction

Start by running examples/sql-to-pandas.py with the reported taxi Parquet data and versions, then inspect the DataFrame.to_pandas entry point shown in the traceback. Compare the schemas of the execution batches, especially the nullable trip_count field, and verify that conversion completes consistently with a pandas DataFrame.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python, rust
Domain
backend, data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.