apache / apache/arrow

`Table.to_pandas()` converts ints to doubles

Open
#43,112 1 comment 0 reactions 1 assignee Claimed by @attwelveDev View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

When you call `to_pandas`, Arrow converts ints to doubles This leads to precision issues (e.g., some ints can't be represented with doubles).

We can potentially avoid this issue by using the pandas nullable integer data type: https://pandas.pydata.org/docs/user_guide/integer_na.html.

```python
import pyarrow

table = pyarrow.Table.from_pydict({"column": [0, None]})
df = table.to_pandas()
assert df.dtypes[0] == int, df.dtypes[0]
```

```
Traceback (most recent call last):
File "/Users/balaji/Documents/GitHub/ray/1.py", line 5, in
assert df.dtypes[0] == int, df.dtypes[0]
^^^^^^^^^^^^^^^^^^^
AssertionError: float64
```

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.