ArrowInvalid Error when converting DataFrame to pandas
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Describe the bug
I am testing the example from [sql-to-pandas.py](https://github.com/apache/datafusion-python/blob/main/examples/sql-to-pandas.py) and I got en error ArrowInvalid with different schema on the line: pandas_df = df.to_pandas().
ArrowInvalid: Schema at index 0 was different:
passenger_count: int64
trip_count: int32 not null
vs
passenger_count: int64
trip_count: int32
Is there any way how to define trip_acount as not nullable when running select or how to make pandas to accept such a DataFrame.
I am testing with conda env with python 3.11.9 on windows. Pip packages I have listed bellow.
```
---------------------------------------------------------------------------
ArrowInvalid Traceback (most recent call last)
Cell In[11], [line 19](vscode-notebook-cell:?execution_count=11&line=19)
[17](vscode-notebook-cell:?execution_count=11&line=17) print(df.schema())
[18](vscode-notebook-cell:?execution_count=11&line=18) # convert to Pandas
---> [19](vscode-notebook-cell:?execution_count=11&line=19) pandas_df = df.to_pandas()
[21](vscode-notebook-cell:?execution_count=11&line=21) # create a chart
[22](vscode-notebook-cell:?execution_count=11&line=22) fig = pandas_df.plot(kind="bar", title="Trip Count by Number of Passengers").get_figure()
File c:\Users\LDolezal\Miniconda3\envs\pyarrow\Lib\site-packages\datafusion\dataframe.py:472, in DataFrame.to_pandas(self)
[466](file:///C:/Users/LDolezal/Miniconda3/envs/pyarrow/Lib/site-packages/datafusion/dataframe.py:466) def to_pandas(self) -> pd.DataFrame:
[467](file:///C:/Users/LDolezal/Miniconda3/envs/pyarrow/Lib/site-packages/datafusion/dataframe.py:467) """Execute the :py:class:`DataFrame` and convert it into a Pandas DataFrame.
[468](file:///C:/Users/LDolezal/Miniconda3/envs/pyarrow/Lib/site-packages/datafusion/dataframe.py:468)
[469](file:///C:/Users/LDolezal/Miniconda3/envs/pyarrow/Lib/site-packages/datafusion/dataframe.py:469) Returns:
[470](file:///C:/Users/LDolezal/Miniconda3/envs/pyarrow/Lib/site-packages/datafusion/dataframe.py:470) Pandas DataFrame.
[471](file:///C:/Users/LDolezal/Miniconda3/envs/pyarrow/Lib/site-packages/datafusion/dataframe.py:471) """
--> [472](file:///C:/Users/LDolezal/Miniconda3/envs/pyarrow/Lib/site-packages/datafusion/dataframe.py:472) return self.df.to_pandas()
File c:\Users\LDolezal\Miniconda3\envs\pyarrow\Lib\site-packages\pyarrow\table.pxi:4865, in pyarrow.lib.Table.from_batches()
File c:\Users\LDolezal\Miniconda3\envs\pyarrow\Lib\site-packages\pyarrow\error.pxi:155, in pyarrow.lib.pyarrow_internal_check_status()
File c:\Users\LDolezal\Miniconda3\envs\pyarrow\Lib\site-packages\pyarrow\error.pxi:92, in pyarrow.lib.check_status()
ArrowInvalid: Schema at index 0 was different:
passenger_count: int64
trip_count: int32 not null
vs
passenger_count: int64
trip_count: int32
```
Data source for the data is here:
https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2024-01.parquet
### To Reproduce
```
from datafusion import SessionContext
ctx = SessionContext()
# Register table with context
ctx.register_parquet('taxi', 'datasets/yellow_tripdata_2024-01.parquet')
# Execute SQL
df = ctx.sql("select passenger_count, cast(count(*) as int) as trip_count "
"from taxi "
"where passenger_count is not null "
"group by passenger_count "
"order by passenger_count")
# convert to Pandas
df.show()
print(df.schema())
# convert to Pandas
pandas_df = df.to_pandas()
# create a chart
fig = pandas_df.plot(kind="bar", title="Trip Count by Number of Passengers").get_figure()
fig.savefig('chart.png')
```
### Expected behavior
_No response_
### Additional context
Package Version
----------------- -----------
asttokens 2.0.5
colorama 0.4.6
comm 0.2.1
datafusion 40.1.0
debugpy 1.6.7
decorator 5.1.1
executing 0.8.3
ipykernel 6.28.0
ipython 8.25.0
jedi 0.19.1
jupyter_client 8.6.0
jupyter_core 5.7.2
matplotlib-inline 0.1.6
nest-asyncio 1.6.0
numpy 2.1.1
packaging 24.1
pandas 2.2.2
parso 0.8.3
pip 24.2
platformdirs 3.10.0
prompt-toolkit 3.0.43
psutil 5.9.0
pure-eval 0.2.2
pyarrow 17.0.0
Pygments 2.15.1
python-dateutil 2.9.0.post0
pytz 2024.1
pywin32 305.1
pyzmq 25.1.2
setuptools 72.1.0
six 1.16.0
stack-data 0.2.0
tornado 6.4.1
traitlets 5.14.3
typing_extensions 4.11.0
tzdata 2024.1
wcwidth 0.2.5
wheel 0.43.0
Contributor guide
Research direction
Start by running examples/sql-to-pandas.py with the reported taxi Parquet data and versions, then inspect the DataFrame.to_pandas entry point shown in the traceback. Compare the schemas of the execution batches, especially the nullable trip_count field, and verify that conversion completes consistently with a pandas DataFrame.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python, rust
- Domain
- backend, data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100