Casting a decimal column of a table into a float type when converting a table into a pandas dataframe
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
Hey All,
I was wondering what is the most elegant & efficient way of casting a decimal column of a table into a float type, when converting a table into a pandas dataframe?
In the code I'm working on, I got to a situation which a `Table` instance has a column of a `decimal128(6, 5)` data type.
I'd like to convert that into a `float` data type.
I tried the following:
(see [this](https://arrow.apache.org/docs/python/api/datatypes.html#type-checking) & [this](https://arrow.apache.org/docs/python/pandas.html#nullable-types) for reference)
```python
import pandas as pd
import pyarrow as pa
def converter(data_type):
if pa.types.is_decimal(data_type):
return pd.Float64Dtype()
return None
table = ... # contains a decimal column
df = table.to_pandas(types_mapper=converter)
```
But when I tried it I got the following error: `TypeError: Expected array of Float64 type, got decimal128(6, 5) instead`
Can anyone please suggest a better way to cast such column into a float data type?
Thanks in advance!
Contributor guide
Research direction
Start by reproducing the failure through Table.to_pandas(types_mapper=converter) with a decimal128(6, 5) column, then inspect the pandas conversion path and its handling of types_mapper. Done means decimal columns can be converted to a pandas float type without the reported TypeError, with regression coverage for this case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100