Refactor `Scan` IO nodes in cudf-polars
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
Currently, we translate all polars `Scan` nodes into our `Scan` type: https://github.com/rapidsai/cudf/blob/11af57b0fbc8eba8f3e825acb517550b71d00817/python/cudf_polars/cudf_polars/dsl/translate.py#L366-L367
Scans can come in various `typ`s, one per format (parquet, csv, ndjson, etc.) By putting everything into a single `Scan` type, we end up with parquet-specific arguments (e.g. `parquet_options`) and methods in a generic base class (and https://github.com/rapidsai/cudf/pull/22700 is making this worse, but adding more parquet specific options).
We should explore splitting this `Scan` node into multiple types, perhaps `ParquetScan`, which is the most custom, and leave `Scan` for the remainder.
When we do this, we should ensure that `typ` is a `Literal` and that we can clearly reason about which scan types are allowed in which places (e.g. which scan types do `SplitScan`, `FusedScan`, and `StreamingScan` support?).
Contributor guide
Research direction
Start in python/cudf_polars/cudf_polars/dsl/translate.py at the Scan translation referenced in the issue, then trace the Scan, SplitScan, FusedScan, and StreamingScan types. Separate parquet-specific behavior from the generic scan representation, make typ a Literal, and document or enforce which scan types each consumer supports.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100