apache / apache/arrow

[Python] `ParquetFileFragment.subset` filter ignored when chained with `to_batches`

Open
#43,274 0 comments 0 reactions 0 assignees View on GitHub
Component: Parquet Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

Based on the [doc](https://arrow.apache.org/docs/python/generated/pyarrow.dataset.ParquetFileFragment.html#pyarrow.dataset.ParquetFileFragment.subset), `ParquetFileFragment.subset` creates a subset of the fragment and returns the same type `ParquetFileFragment` as output. However, when I continue to iterate through the fragment via `to_batches`, I found that the subset filter is not applied. Whether the `subset` filtering is implemented eagerly or lazily, I expect to be able to chain further operations like `to_batches` after it, since the output type is still a `ParquetFileFragment`.

Here's a snippet to reproduce the behavior I mentioned. You would need a `test.parquet` containing a `character` column with values like 'a', 'b', 'c', 'd'. After running this script, you should see: `Characters in unique_chars but not in charset: d`.

```python
import pyarrow.parquet as pq
import pyarrow.compute as pc

charset = ['a', 'b', 'c']

filter = pc.field('character').isin(charset)

dataset = pq.ParquetDataset('test.parquet', filters=filter)

unique_chars = set()

for fragment in dataset.fragments:
# fragment = fragment.subset(filter)
for batch in fragment.to_batches():
df = batch.to_pandas()
for character in df['character']:
unique_chars.add(character)

diff_chars = unique_chars.difference(charset)
print("Characters in unique_chars but not in charset:", diff_chars)
```

### Component(s)

Parquet, Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.