NVIDIA / NVIDIA/cudf

resolve engine-bootstrap options once, at construction, and freeze them

Open
#23,745 0 comments 0 reactions 1 assignee Claimed by @Matt711 View on GitHub
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

Part of #23740.

`SPMDEngine`/`DaskEngine`/`RayEngine.__init__` need `num_py_executors`,
`hardware_binding`, `memory_resource_config`, `quent_context`, `kvikio_nthreads`, and
`allow_gpu_sharing` before `ConfigOptions`/`StreamingExecutor` can exist, to actually
build a thread pool or bind a GPU. Build one `StreamingOptions` at construction, read
these off it, and write the fully-resolved dict back as `executor_options`/
`engine_options` before calling `super().__init__`, so `self.config` is a frozen, correct
snapshot from the moment the engine exists instead of a partially-resolved one
`ConfigOptions.from_polars_engine` has to patch up on every query. Once every field
resolves to a concrete value this way, `_reset()` can just merge the caller's overrides
onto the previous config instead of needing per-field carry-forward logic.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.