NVIDIA / NVIDIA/cudf

[Story]: Rework how cudf-polars resolves executor/engine configuration

Open
#23,740 2 comments 0 reactions 0 assignees View on GitHub
cudf-polars improvement
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

cudf-polars resolves executor/engine config (explicit value, then env var, then default)
in three separate places today: `StreamingOptions`, `StreamingExecutor`/
`ConfigOptions.from_polars_engine`, and a handful of ad hoc reads inside
`SPMDEngine`/`DaskEngine`/`RayEngine.__init__`. The same env var name and default end up
written two or three times for the same field, with nothing keeping them in sync. Some
fields end up reading a hardcoded fallback instead of the actual env var because one of
the duplicated copies never checks it, and `_reset()` doesn't consistently preserve a
field's previous value across an unrelated reset.

We want one shared way to resolve a field's value regardless of which class declares it,
only deferring resolution for fields that truly need it, and resolving an engine's config
once at construction instead of on every query.

Contributor guide

Open the contributing guide

Research direction

Start by tracing configuration resolution through StreamingOptions, StreamingExecutor, ConfigOptions.from_polars_engine, and the SPMDEngine, DaskEngine, and RayEngine constructors. Compare explicit values, environment variables, defaults, and _reset() behavior across these entry points. Done means shared resolution is consistent, deferred only where needed, and each engine resolves its configuration once at construction.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.