[Python] Table.from_pandas is not consistent when inferring the schema from a Pandas dataframe
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
I have pandas Dataframes that can contain lists of floats that sometimes can be NaNs. If all the entries are NaNs `pa.Table.from_pandas` infers a list but this only happens if it's all NaNs. If one entry is a regular float it becomes list (which is what I expect). Interesting if you have a list of np.float32 then you get a list even if it's all NaNs. See:
```import numpy as np
import pandas as pd
import pyarrow as pa
df = pd.DataFrame(
{
"col": [[np.nan]],
}
)
print(pa.Table.from_pandas(df).schema.field(0).type)
df = pd.DataFrame(
{
"col": [[np.float64(np.nan)]],
}
)
print(pa.Table.from_pandas(df).schema.field(0).type)
df = pd.DataFrame(
{
"col": [[np.float32(np.nan)]],
}
)
print(pa.Table.from_pandas(df).schema.field(0).type)
df = pd.DataFrame(
{
"col": [[1.0, np.nan]],
}
)
print(pa.Table.from_pandas(df).schema.field(0).type)
```
Is there a way to enforce that I get always list as technically isinstance(np.nan, float) is True.
thanks
cc @alanhdu
### Component(s)
Python
Contributor guide
Assessment
This issue has not been assessed yet.