apache / apache/datafusion

[Python] Converting a dataframe to a python list results in a higher precision number

Open
#9,950 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

Converting a dataframe into a python list, makes the result a higher precision (float32 to float64)

### To Reproduce

```
import datafusion
import pyarrow as pa
ctx = datafusion.SessionContext()
batch = pa.RecordBatch.from_arrays([pa.array([2.55], pa.float32())], names=['arg0'])
ctx.register_record_batches("my_table", [[batch]])
expr = 'SELECT abs(arg0) FROM my_table;'
ctx.sql(expr)
DataFrame()
+--------------------+
| abs(my_table.arg0) |
+--------------------+
| 2.55 |
+--------------------+
ctx.sql(expr).collect()[0].columns[0].to_pylist()
[2.549999952316284]
```

### Expected behavior

My expectation was that I should still get 2.55

### Additional context

Seems like the python standard float is a C double

Contributor guide

Open the contributing guide

Research direction

Start at the Python-facing path exercised by ctx.sql(expr).collect()[0].columns[0].to_pylist(), using the provided float32 PyArrow RecordBatch reproduction. Determine how conversion should preserve the reported float32 behavior, then verify that the returned list matches the expected result.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.