NVIDIA / NVIDIA/cudf

`Select(len())` fast path doesn't cover `DataFrameScan`

Open
#24,112 0 comments 0 reactions 0 assignees View on GitHub
bug cudf-polars
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

`Select.evaluate` (`ir.py`) has a fast-count pushdown for `Select(len())` over a `Scan(parquet)` — it reads the row count from file metadata instead of actually scanning. `DataFrameScan.do_evaluate` already computes `height = pl_df.height` for free, and even has a zero-width-schema fast path that returns using only `height`. But for any non-zero-width schema, it always falls through to `DataFrame.from_polars`, materializing the whole frame on GPU even when the caller only wants `len()`.

This isn't just a missed optimization, it can crash. polars' `Series.new_from_index` builds a virtual, unmaterialized series of arbitrary length (no real backing data). Union two of those and do `.select(pl.len())`, and cudf-polars tries to materialize both branches for real, hitting libcudf's 32-bit `size_type` limit: `OverflowError: ... Number of rows exceeds cuDF's maximum supported row count (cudf::size_type)`. Under the streaming engine this actually crashes the worker instead of raising cleanly.

Fix would be extending the existing `Select._is_len_expr` fast path past `Scan(parquet)` to also recognize `DataFrameScan` (using `pl_df.height`), and see through `Cache`/`Union` so branch heights can be summed instead of materialized.

Repro is upstream polars' `tests/unit/lazyframe/test_projections.py::test_projection_pushdown_union_len_pushdown_28657`, currently xfailed/skipped in cudf-polars' `inject_gpu_engine.py`.

Contributor guide

Open the contributing guide

Research direction

Start in ir.py at Select._is_len_expr and DataFrameScan.do_evaluate, comparing the existing Scan(parquet) fast path with DataFrameScan's height handling. Trace Cache and Union behavior, then run tests/unit/lazyframe/test_projections.py::test_projection_pushdown_union_len_pushdown_28657 and review its xfail/skip in inject_gpu_engine.py. Done means len() uses branch heights without materializing non-empty frames and the regression no longer needs to be skipped.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
data, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.