apache / apache/arrow

Casting a decimal column of a table into a float type when converting a table into a pandas dataframe

Open
#13,353 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

Hey All,

I was wondering what is the most elegant & efficient way of casting a decimal column of a table into a float type, when converting a table into a pandas dataframe?

In the code I'm working on, I got to a situation which a `Table` instance has a column of a `decimal128(6, 5)` data type.
I'd like to convert that into a `float` data type.

I tried the following:
(see [this](https://arrow.apache.org/docs/python/api/datatypes.html#type-checking) & [this](https://arrow.apache.org/docs/python/pandas.html#nullable-types) for reference)

```python
import pandas as pd
import pyarrow as pa

def converter(data_type):
if pa.types.is_decimal(data_type):
return pd.Float64Dtype()
return None

table = ... # contains a decimal column
df = table.to_pandas(types_mapper=converter)
```

But when I tried it I got the following error: `TypeError: Expected array of Float64 type, got decimal128(6, 5) instead`

Can anyone please suggest a better way to cast such column into a float data type?

Thanks in advance!

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the failure through Table.to_pandas(types_mapper=converter) with a decimal128(6, 5) column, then inspect the pandas conversion path and its handling of types_mapper. Done means decimal columns can be converted to a pandas float type without the reported TypeError, with regression coverage for this case.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.