[Python] Converting a dataframe to a python list results in a higher precision number
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Describe the bug
Converting a dataframe into a python list, makes the result a higher precision (float32 to float64)
### To Reproduce
```
import datafusion
import pyarrow as pa
ctx = datafusion.SessionContext()
batch = pa.RecordBatch.from_arrays([pa.array([2.55], pa.float32())], names=['arg0'])
ctx.register_record_batches("my_table", [[batch]])
expr = 'SELECT abs(arg0) FROM my_table;'
ctx.sql(expr)
DataFrame()
+--------------------+
| abs(my_table.arg0) |
+--------------------+
| 2.55 |
+--------------------+
ctx.sql(expr).collect()[0].columns[0].to_pylist()
[2.549999952316284]
```
### Expected behavior
My expectation was that I should still get 2.55
### Additional context
Seems like the python standard float is a C double
Contributor guide
Research direction
Start at the Python-facing path exercised by ctx.sql(expr).collect()[0].columns[0].to_pylist(), using the provided float32 PyArrow RecordBatch reproduction. Determine how conversion should preserve the reported float32 behavior, then verify that the returned list matches the expected result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100