`Select(len())` fast path doesn't cover `DataFrameScan`
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
`Select.evaluate` (`ir.py`) has a fast-count pushdown for `Select(len())` over a `Scan(parquet)` — it reads the row count from file metadata instead of actually scanning. `DataFrameScan.do_evaluate` already computes `height = pl_df.height` for free, and even has a zero-width-schema fast path that returns using only `height`. But for any non-zero-width schema, it always falls through to `DataFrame.from_polars`, materializing the whole frame on GPU even when the caller only wants `len()`.
This isn't just a missed optimization, it can crash. polars' `Series.new_from_index` builds a virtual, unmaterialized series of arbitrary length (no real backing data). Union two of those and do `.select(pl.len())`, and cudf-polars tries to materialize both branches for real, hitting libcudf's 32-bit `size_type` limit: `OverflowError: ... Number of rows exceeds cuDF's maximum supported row count (cudf::size_type)`. Under the streaming engine this actually crashes the worker instead of raising cleanly.
Fix would be extending the existing `Select._is_len_expr` fast path past `Scan(parquet)` to also recognize `DataFrameScan` (using `pl_df.height`), and see through `Cache`/`Union` so branch heights can be summed instead of materialized.
Repro is upstream polars' `tests/unit/lazyframe/test_projections.py::test_projection_pushdown_union_len_pushdown_28657`, currently xfailed/skipped in cudf-polars' `inject_gpu_engine.py`.
Contributor guide
Research direction
Start in ir.py at Select._is_len_expr and DataFrameScan.do_evaluate, comparing the existing Scan(parquet) fast path with DataFrameScan's height handling. Trace Cache and Union behavior, then run tests/unit/lazyframe/test_projections.py::test_projection_pushdown_union_len_pushdown_28657 and review its xfail/skip in inject_gpu_engine.py. Done means len() uses branch heights without materializing non-empty frames and the regression no longer needs to be skipped.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- data, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100