[Python] Round-trip type-conversion bug list object in pd.DataFrame
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
### Describe the bug, including details regarding any error messages, version, and platform.
I am trying to convert a pandas DataFrame into a table and back. However the type property is lost, the list is returned as an numpy.ndarray
```
import pandas as pd
import pyarrow as pa
df = pd.DataFrame(dict(a=[[1,2,3],]))
type(df['a'].iloc[0])
# Out[1]: list
round_trip_df = pa.Table.from_pandas(df).to_pandas()
type(round_trip_df['a'].iloc[0])
# Out[2]: numpy.ndarray
```
Here is the output of `pd.show_versions()`;
INSTALLED VERSIONS
------------------
commit : 2e218d10984e9919f0296931d92ea851c6a6faf5
python : 3.10.9.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19045
machine : AMD64
processor : Intel64 Family 6 Model 142 Stepping 12, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : English_United States.1252
pandas : 1.5.3
numpy : 1.23.5
pytz : 2022.7.1
dateutil : 2.8.2
setuptools : 64.0.2
pip : 23.0
Cython : 0.29.33
pytest : 7.2.1
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.9.2
html5lib : 1.1
pymysql : None
psycopg2 : 2.9.3
jinja2 : 3.1.2
IPython : 8.10.0
pandas_datareader: 0.10.0
bs4 : 4.11.2
bottleneck : None
brotli :
fastparquet : None
fsspec : 2023.1.0
gcsfs : None
matplotlib : 3.7.0
numba : 0.56.4
numexpr : None
odfpy : None
openpyxl : 3.1.1
pandas_gbq : None
pyarrow : 9.0.0
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.8.1
snappy : None
sqlalchemy : 1.4.46
tables : None
tabulate : 0.9.0
xarray : None
xlrd : None
xlwt : None
zstandard : None
tzdata : None
### Component(s)
Python
Contributor guide
Research direction
Start by running the reported pandas DataFrame round trip through pyarrow.Table.from_pandas().to_pandas() and confirm that a list-valued cell becomes a numpy.ndarray. Trace the Python conversion entry points involved and find the corresponding regression-test area. Done means the round trip preserves list values while existing conversions remain passing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, pandas, python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100