NVIDIA / NVIDIA/cudf

Refactor `Scan` IO nodes in cudf-polars

Open
#23,063 0 comments 0 reactions 0 assignees View on GitHub
cudf-polars
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

Currently, we translate all polars `Scan` nodes into our `Scan` type: https://github.com/rapidsai/cudf/blob/11af57b0fbc8eba8f3e825acb517550b71d00817/python/cudf_polars/cudf_polars/dsl/translate.py#L366-L367

Scans can come in various `typ`s, one per format (parquet, csv, ndjson, etc.) By putting everything into a single `Scan` type, we end up with parquet-specific arguments (e.g. `parquet_options`) and methods in a generic base class (and https://github.com/rapidsai/cudf/pull/22700 is making this worse, but adding more parquet specific options).

We should explore splitting this `Scan` node into multiple types, perhaps `ParquetScan`, which is the most custom, and leave `Scan` for the remainder.

When we do this, we should ensure that `typ` is a `Literal` and that we can clearly reason about which scan types are allowed in which places (e.g. which scan types do `SplitScan`, `FusedScan`, and `StreamingScan` support?).

Contributor guide

Open the contributing guide

Research direction

Start in python/cudf_polars/cudf_polars/dsl/translate.py at the Scan translation referenced in the issue, then trace the Scan, SplitScan, FusedScan, and StreamingScan types. Separate parquet-specific behavior from the generic scan representation, make typ a Literal, and document or enforce which scan types each consumer supports.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.