apache / apache/arrow

[Python] Dataset.to_table filter error

Open
#36,719 4 comments 0 reactions 0 assignees View on GitHub
Component: Python
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

We faced an issue while trying to use row filter in `Dataset.to_table`:

```
import pyarrow.dataset as ds
import pyarrow as pa
import pyarrow.parquet as pq

amount = pa.array([9999999, 100000001])

table = pa.Table.from_arrays([amount], names=["amount"])
pq.write_table(table, 'mre.parquet')

table = ds.dataset('mre.parquet').to_table(filter=ds.field('amount') / 1e8 > 1) # fails
table = ds.dataset('mre.parquet').to_table(filter=ds.field('amount') / 100000000 > 1) # works
```

Error:
```
Traceback (most recent call last):
File "/Users/dytyniak/projects/my_project/local_utils/testing.py", line 29, in
table = ds.dataset('mre.parquet').to_table(filter=ds.field('amount') / 1e8 > 1)
File "pyarrow/_dataset.pyx", line 546, in pyarrow._dataset.Dataset.to_table
File "pyarrow/_dataset.pyx", line 3449, in pyarrow._dataset.Scanner.to_table
File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Integer value 100000001 not in range: -16777216 to 16777216
```

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.