[C++] get_fragments filter argument not filtering on statistics
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
### Describe the bug, including details regarding any error messages, version, and platform.
Based on the documentation
https://arrow.apache.org/docs/6.0/python/generated/pyarrow.dataset.Dataset.html#pyarrow.dataset.Dataset.get_fragments
_Return fragments matching the optional filter, either using the partition_expression or internal information like Parquet’s statistics._
I would have assumed that the following code would return only one fragment. However, the expression seems to be applied only to the partitions as both fragments are being returned even though only one matches the predicate if you look at the statitics
```python
import pyarrow.dataset as pds
import pandas as pd
ds_path = './my_dataset'
df = pd.DataFrame({
'A': [1,2],
'B': [1,2]
})
df.to_parquet(ds_path, partition_cols=['A'])
pdataset = pds.dataset(ds_path, format='parquet', partitioning='hive')
fragments = pdataset.get_fragments(
filter=(pds.field('B') == 1)
)
for f in fragments:
print('-------------')
print(f.path)
print(f.to_table())
for row_group in f.row_groups:
print(row_group.statistics)
```
### Version
'11.0.0'
### Component(s)
Python
Contributor guide
Research direction
Start at pyarrow.dataset.Dataset.get_fragments and trace how its filter reaches fragment discovery; the issue names no source file or test. Reproduce the supplied pandas/Parquet example, then add regression coverage showing that Parquet statistics are honored when filtering fragments.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100