[Python] Dataset.to_table filter error
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
We faced an issue while trying to use row filter in `Dataset.to_table`:
```
import pyarrow.dataset as ds
import pyarrow as pa
import pyarrow.parquet as pq
amount = pa.array([9999999, 100000001])
table = pa.Table.from_arrays([amount], names=["amount"])
pq.write_table(table, 'mre.parquet')
table = ds.dataset('mre.parquet').to_table(filter=ds.field('amount') / 1e8 > 1) # fails
table = ds.dataset('mre.parquet').to_table(filter=ds.field('amount') / 100000000 > 1) # works
```
Error:
```
Traceback (most recent call last):
File "/Users/dytyniak/projects/my_project/local_utils/testing.py", line 29, in
table = ds.dataset('mre.parquet').to_table(filter=ds.field('amount') / 1e8 > 1)
File "pyarrow/_dataset.pyx", line 546, in pyarrow._dataset.Dataset.to_table
File "pyarrow/_dataset.pyx", line 3449, in pyarrow._dataset.Scanner.to_table
File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Integer value 100000001 not in range: -16777216 to 16777216
```
### Component(s)
Python
Contributor guide
Assessment
This issue has not been assessed yet.