apache / apache/arrow

[C++] get_fragments filter argument not filtering on statistics

Open
#35,841 1 comment 0 reactions 0 assignees View on GitHub
Component: C++ Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

### Describe the bug, including details regarding any error messages, version, and platform.

Based on the documentation
https://arrow.apache.org/docs/6.0/python/generated/pyarrow.dataset.Dataset.html#pyarrow.dataset.Dataset.get_fragments
_Return fragments matching the optional filter, either using the partition_expression or internal information like Parquet’s statistics._

I would have assumed that the following code would return only one fragment. However, the expression seems to be applied only to the partitions as both fragments are being returned even though only one matches the predicate if you look at the statitics

```python
import pyarrow.dataset as pds
import pandas as pd

ds_path = './my_dataset'
df = pd.DataFrame({
'A': [1,2],
'B': [1,2]
})
df.to_parquet(ds_path, partition_cols=['A'])

pdataset = pds.dataset(ds_path, format='parquet', partitioning='hive')

fragments = pdataset.get_fragments(
filter=(pds.field('B') == 1)
)

for f in fragments:
print('-------------')
print(f.path)
print(f.to_table())
for row_group in f.row_groups:
print(row_group.statistics)
```
### Version
'11.0.0'

### Component(s)

Python

Contributor guide

Open the contributing guide

Research direction

Start at pyarrow.dataset.Dataset.get_fragments and trace how its filter reaches fragment discovery; the issue names no source file or test. Reproduce the supplied pandas/Parquet example, then add regression coverage showing that Parquet statistics are honored when filtering fragments.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.