apache / apache/arrow

[Python] PyArrow Table to Pandas int8 conversion issue

Open
#40,815 3 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

Hi team,

We extensively use Arrow / Parquet files for data analysis with Pandas, it's excellent! We've found an issue that occurs converting between PyArrow Tables and Pandas Dataframe.

Due to large size of our dataset we write the parquet files using minimal field data types e.g categories and specifically we're using an `int8` in `field_6`. This field is unfortunately nullable which is likely the cause of the issue.

The issue is that the `int8` field is converted to Pandas as a `float64`. In Pandas this can be fixed using the `dtype_backend="numpy_nullable"` argument and it converts to an `Int8`.

Is there any equivalent mechanism, using pyarrow.parquet.read_pandas to retrieve the field with an `Int8` or equivalent? I assume the conversion to `float64` is forced due to the field being nullable.

Many thanks!

PyArrow Parquet Metadata:
```

created_by: parquet-cpp-arrow version 15.0.2
num_columns: 11
num_rows: 35734802
num_row_groups: 35
format_version: 2.6
serialized_size: 41957
```

PyArrow Parquet Schema:
```

required group field_id=-1 schema {
optional int64 field_id=-1 timestamp (Timestamp(isAdjustedToUTC=false, timeUnit=milliseconds, is_from_converted_type=false, force_set_converted_type=false));
optional binary field_id=-1 field_1 (String);
optional binary field_id=-1 field_2 (String);
optional binary field_id=-1 field_3 (String);
optional binary field_id=-1 field_4 (String);
optional binary field_id=-1 field_5 (String);
optional int32 field_id=-1 field_6 (Int(bitWidth=8, isSigned=true)); <------ note Int8
optional double field_id=-1 field_7;
optional double field_id=-1 field_8;
optional double field_id=-1 field_9;
optional double field_id=-1 field_10;
}
```

1. Pandas read parquet:
```
df = pd.read_parquet(filename)
timestamp datetime64[ms]
field_1 category
field_2 category
field_3 category
field_4 category
field_5 category
field_6 float64 <---- note float64
field_7 float64
field_8 float64
field_9 float64
field_10 float64
```

2. Pandas read parquet + numpy_nullable:
```
df = pd.read_parquet(filename, dtype_backend="numpy_nullable")
timestamp datetime64[ms]
field_1 category
field_2 category
field_3 category
field_4 category
field_5 category
field_6 Int8 <---- note Int8
field_7 Float64
field_8 Float64
field_9 Float64
field_10 Float64
```

3. PyArrow parquet read table:
```
table = pyarrow.parquet.read_pandas(target_path, filesystem=s3_client)
df = table.to_pandas()
timestamp datetime64[ms]
field_1 category
field_2 category
field_3 category
field_4 category
field_5 category
field_6 float64 <---- note float64
field_7 float64
field_8 float64
field_9 float64
field_10 float64
```

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.