apache / apache/arrow-rs

[parquet] Extend reader fuzz coverage to predicate pushdown, predicate caching, and row selection policy

Open
#10,747 1 comment 0 reactions 1 assignee Claimed by @fyrsta7 View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 16h
Merged PRs (30d)
168

Description

### Is your feature request related to a problem or challenge?

#10733 (fixed by #10735) needed four things to be true at the same time: predicate pushdown, page pruning, a Mask-backed `RowSelection`, and the predicate cache. Each of those is covered by tests today. No test covered any two of them together, which is why it shipped in 59.2.0 and survived about two weeks before anyone hit it.

Concretely, the two relevant test areas are disjoint:

| test area | `RowSelectionPolicy` | `with_row_selection` | `PageIndexPolicy` | predicate cache |
|---|---|---|---|---|
| `parquet/tests/arrow_reader/predicate_cache.rs` | 0 | 0 | 0 | yes |
| mask/sparse tests (`read_plan.rs`, `push_decoder/reader_builder/mod.rs`, `row_filter/async.rs`) | yes | yes | yes | **0** |

This is a gap in the cross product rather than a gap in depth: #10288 added a fair number of unit tests for the mask/sparse-page interaction, and they all pass on the buggy code.

### Describe the solution you'd like

`test_fuzz_async_reader_selection` in `parquet/src/arrow/async_reader/mod.rs` is the natural home, but it cannot reach this class of bug as written. Today it randomizes only the selection pattern and one projected column, and it has **no `RowFilter` at all** — so predicate pushdown, the predicate cache, and filter-driven sparse fetch are all unreachable from it.

Axes worth adding:

- [ ] a `RowFilter` (this is the entry condition for the whole path)
- [ ] predicate cache size: disabled / small enough to evict / large
- [ ] `RowSelectionPolicy`: `Selectors` / `Mask` / `Auto`
- [ ] `batch_size` (batch alignment is central to the cached reader)
- [ ] projections that overlap the predicate columns, so a column is both cached and part of the output
- [ ] page size / row group size, so page boundaries and batch boundaries interleave differently

Two other changes would raise the yield:

1. **Stronger oracle.** The current assertion is `assert_eq!(actual_rows, expected_rows)` — row counts only. A read that returns the wrong *values* passes. A differential check against a reference configuration (same selection under `Selectors`, or with the cache disabled) would catch that and needs no golden data.

2. **Better selection generator.** The current generator alternates select/skip with `row_count` drawn from `1..100`, so it essentially never produces the shape that broke here: a single selected row followed by a long skip (`select(1), skip(39), select(1)`). Worth explicitly generating sparse single-row islands, runs that straddle a page boundary, and selections whose gaps are larger than one page.

### Describe alternatives you've considered

End-to-end coverage against a real engine (filed separately as #10746). That catches a different failure mode — a configuration nobody thought to test — and is complementary rather than an alternative.

### Additional context

The same error class was fixed once before in #9301 (January 2026), which suggests this interaction is worth systematic coverage rather than one more targeted test.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.