[Story]: Rework how cudf-polars resolves executor/engine configuration
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
cudf-polars resolves executor/engine config (explicit value, then env var, then default)
in three separate places today: `StreamingOptions`, `StreamingExecutor`/
`ConfigOptions.from_polars_engine`, and a handful of ad hoc reads inside
`SPMDEngine`/`DaskEngine`/`RayEngine.__init__`. The same env var name and default end up
written two or three times for the same field, with nothing keeping them in sync. Some
fields end up reading a hardcoded fallback instead of the actual env var because one of
the duplicated copies never checks it, and `_reset()` doesn't consistently preserve a
field's previous value across an unrelated reset.
We want one shared way to resolve a field's value regardless of which class declares it,
only deferring resolution for fields that truly need it, and resolving an engine's config
once at construction instead of on every query.
Contributor guide
Research direction
Start by tracing configuration resolution through StreamingOptions, StreamingExecutor, ConfigOptions.from_polars_engine, and the SPMDEngine, DaskEngine, and RayEngine constructors. Compare explicit values, environment variables, defaults, and _reset() behavior across these entry points. Done means shared resolution is consistent, deferred only where needed, and each engine resolves its configuration once at construction.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100