apache / apache/arrow

[Python] Table.from_pandas is not consistent when inferring the schema from a Pandas dataframe

Open
#47,238 0 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

I have pandas Dataframes that can contain lists of floats that sometimes can be NaNs. If all the entries are NaNs `pa.Table.from_pandas` infers a list but this only happens if it's all NaNs. If one entry is a regular float it becomes list (which is what I expect). Interesting if you have a list of np.float32 then you get a list even if it's all NaNs. See:

```import numpy as np
import pandas as pd
import pyarrow as pa

df = pd.DataFrame(
{
"col": [[np.nan]],
}
)
print(pa.Table.from_pandas(df).schema.field(0).type)

df = pd.DataFrame(
{
"col": [[np.float64(np.nan)]],
}
)
print(pa.Table.from_pandas(df).schema.field(0).type)

df = pd.DataFrame(
{
"col": [[np.float32(np.nan)]],
}
)
print(pa.Table.from_pandas(df).schema.field(0).type)

df = pd.DataFrame(
{
"col": [[1.0, np.nan]],
}
)
print(pa.Table.from_pandas(df).schema.field(0).type)
```

Image

Is there a way to enforce that I get always list as technically isinstance(np.nan, float) is True.

thanks

cc @alanhdu

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.