[Python] `ParquetFileFragment.subset` filter ignored when chained with `to_batches`
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
Based on the [doc](https://arrow.apache.org/docs/python/generated/pyarrow.dataset.ParquetFileFragment.html#pyarrow.dataset.ParquetFileFragment.subset), `ParquetFileFragment.subset` creates a subset of the fragment and returns the same type `ParquetFileFragment` as output. However, when I continue to iterate through the fragment via `to_batches`, I found that the subset filter is not applied. Whether the `subset` filtering is implemented eagerly or lazily, I expect to be able to chain further operations like `to_batches` after it, since the output type is still a `ParquetFileFragment`.
Here's a snippet to reproduce the behavior I mentioned. You would need a `test.parquet` containing a `character` column with values like 'a', 'b', 'c', 'd'. After running this script, you should see: `Characters in unique_chars but not in charset: d`.
```python
import pyarrow.parquet as pq
import pyarrow.compute as pc
charset = ['a', 'b', 'c']
filter = pc.field('character').isin(charset)
dataset = pq.ParquetDataset('test.parquet', filters=filter)
unique_chars = set()
for fragment in dataset.fragments:
# fragment = fragment.subset(filter)
for batch in fragment.to_batches():
df = batch.to_pandas()
for character in df['character']:
unique_chars.add(character)
diff_chars = unique_chars.difference(charset)
print("Characters in unique_chars but not in charset:", diff_chars)
```
### Component(s)
Parquet, Python
Contributor guide
Assessment
This issue has not been assessed yet.