[BUG] Parquet row-group stats filter fails for `IS_NULL` on nested-type columns
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Describe the bug**
Passing a filter expression that contains `IS_NULL` or `NOT(IS_NULL(...))` on a struct or list
column to the libcudf parquet reader raises an error:
```
ValueError: CUDF failure at: cpp/src/io/parquet/reader_impl_helpers.cpp:66:
Column chunk with schema index 1 not found in row group
```
**Steps/Code to reproduce bug**
```python
import tempfile, pathlib
import polars as pl
import pylibcudf as plc
from pylibcudf import expressions as plc_expr
with tempfile.TemporaryDirectory() as tmp:
path = pathlib.Path(tmp) / "struct.parquet"
pl.DataFrame({
"s": pl.Series(
[{"a": None, "b": 1}, {"a": 2, "b": 3}],
dtype=pl.Struct({"a": pl.Int64, "b": pl.Int64}),
)
}).write_parquet(path)
is_null = plc_expr.Operation(
plc_expr.ASTOperator.IS_NULL,
plc_expr.ColumnNameReference("s"),
)
opts = plc.io.parquet.ParquetReaderOptions.builder(
plc.io.SourceInfo([path])
).build()
opts.set_filter(is_null)
plc.io.parquet.read_parquet(opts)
```
```
ValueError: CUDF failure at: /home/coder/cudf/cpp/src/io/parquet/reader_impl_helpers.cpp:66: Column chunk with schema index 1 not found in row group
```
**Expected behavior**
Skip statistics-based row-group pruning and fallback to apply predicate as a post-read filter.
**Additional context**
This came up in #23143 which allows `IS_NULL` predicates to be pushed down.
Contributor guide
Research direction
Start with the reproducer and inspect cpp/src/io/parquet/reader_impl_helpers.cpp around line 66, then trace the parquet reader's row-group statistics filtering for nested columns and IS_NULL predicates. Done means the struct-column example no longer raises an error and skips statistics pruning while applying the predicate after reading.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100