NVIDIA / NVIDIA/cudf

[BUG] Parquet row-group stats filter fails for `IS_NULL` on nested-type columns

Open
#23,397 0 comments 0 reactions 0 assignees View on GitHub
bug cuIO libcudf
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Describe the bug**
Passing a filter expression that contains `IS_NULL` or `NOT(IS_NULL(...))` on a struct or list
column to the libcudf parquet reader raises an error:

```
ValueError: CUDF failure at: cpp/src/io/parquet/reader_impl_helpers.cpp:66:
Column chunk with schema index 1 not found in row group
```
**Steps/Code to reproduce bug**
```python
import tempfile, pathlib
import polars as pl
import pylibcudf as plc
from pylibcudf import expressions as plc_expr

with tempfile.TemporaryDirectory() as tmp:
path = pathlib.Path(tmp) / "struct.parquet"
pl.DataFrame({
"s": pl.Series(
[{"a": None, "b": 1}, {"a": 2, "b": 3}],
dtype=pl.Struct({"a": pl.Int64, "b": pl.Int64}),
)
}).write_parquet(path)

is_null = plc_expr.Operation(
plc_expr.ASTOperator.IS_NULL,
plc_expr.ColumnNameReference("s"),
)

opts = plc.io.parquet.ParquetReaderOptions.builder(
plc.io.SourceInfo([path])
).build()
opts.set_filter(is_null)

plc.io.parquet.read_parquet(opts)
```
```
ValueError: CUDF failure at: /home/coder/cudf/cpp/src/io/parquet/reader_impl_helpers.cpp:66: Column chunk with schema index 1 not found in row group
```

**Expected behavior**
Skip statistics-based row-group pruning and fallback to apply predicate as a post-read filter.

**Additional context**
This came up in #23143 which allows `IS_NULL` predicates to be pushed down.

Contributor guide

Open the contributing guide

Research direction

Start with the reproducer and inspect cpp/src/io/parquet/reader_impl_helpers.cpp around line 66, then trace the parquet reader's row-group statistics filtering for nested columns and IS_NULL predicates. Done means the struct-column example no longer raises an error and skips statistics pruning while applying the predicate after reading.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.