apache / apache/gluten

[CORE] Native conf passing is decided by hard-coded string key lists

Open
#12,694 0 comments 0 reactions 1 assignee Claimed by @jackylee-ch View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Description

Which configurations get delivered from JVM to native side is currently decided by hard-coded
string lists inside `GlutenConfig.getNativeSessionConf` / `getNativeBackendConf`:

- a `nativeKeys` set of 40+ raw key strings;
- two "configs having default values" `Seq`s that restate each conf's key **and** its default;
- per-key special cases inlined into the selection methods (byte-unit conversion for
`spark.shuffle.file.buffer`, upper-casing for `spark.sql.legacy.timeParserPolicy`, …);
- `BackendSettingsApi.extraNativeSessionConfKeys` / `extraNativeBackendConfKeys` as the escape
hatch for backends.

#### Why it matters

**The declaration lives far from the definition.** Adding a native conf means editing a central
list in `gluten-substrait`, not the conf's own definition. Nothing links the two, so the usual
failure mode is a conf that looks wired up but never reaches native — or the reverse, a key that
keeps being delivered long after its native reader was removed. `spark.gluten.velox.fs.s3a.retry.mode`
is an instance of the latter: it has had no native reader since #8123 moved the S3 config path to
Velox's `S3Config`, yet it is still on the list.

**Defaults are duplicated and can drift.** The default-value `Seq`s repeat what the `ConfigEntry`
already declares. Two copies of the same default, updated by hand.

**Backend-specific keys live in common code.** `gluten-substrait` enumerates Velox S3 keys, so a
ClickHouse-only deployment carries Velox key names, and vice versa. There is no module boundary.

**Delivery scope is invisible at the definition.** A conf is delivered on the backend channel, the
runtime channel, or both purely by virtue of which list it appears in. Reading the conf's
definition tells you nothing about it, and a mismatch between the declared mutability of a conf and
where native actually consumes it is impossible to notice. Surveying the current lists turned up
several: `velox.cudf.enableTableScan` is declared static yet read per query;
`memoryOverhead.size.in.bytes` is declared modifiable yet consumed only at backend init.

**Third parties cannot participate.** An out-of-tree backend or component has no way to declare its
own native confs. `extraNativeBackendConfKeys` is the only hook, it takes a bare `Set[String]`
(no per-channel scope, no default, no normalization), and no backend in the tree has ever
overridden it.

#### Proposal

Move the declaration to each conf's definition, and let the conf's own properties decide how it is
delivered.

1. **Declare at the definition site.** A marker on the config builder, e.g. `passToNative()`,
states that the conf reaches native. Related markers cover the two remaining behaviors the old
per-key special cases provided: delivering the conf's default when the user did not set it (for
keys native relies on being present), and normalizing a value before delivery.

2. **Derive the channel from mutability rather than stating it.** A conf that is modifiable at any
time should be delivered wherever native might read it — both at backend initialization and on
each native runtime creation. A conf that is set at backend init and immutable afterwards only
needs delivering once, at backend init. This makes the channel a consequence of a property the
declaration already states, so there is nothing extra to get wrong, and it forces the
mutability/consumption mismatches above to be resolved rather than papered over.

3. **Support Spark- and Hadoop-owned keys.** Keys such as `spark.sql.orc.compression.codec` or
`spark.hadoop.input.read.timeout` must **not** get a Gluten `ConfigEntry` — their owner already
registered them, and registering again conflicts with `SQLConf`. They need a declaration form
that states only the native delivery.

4. **Express "overrides a Spark conf" declaratively.** Several Gluten confs exist only to depart
from a Spark default (`spark.gluten.sql.columnar.shuffle.codec` over
`spark.io.compression.codec`). Today the relationship is hand-written at each read site. A
fallback declaration should carry it, including whether a value came from the Gluten key or was
inherited — callers that validate the two differently need that distinction.

5. **Give components an entry point.** A conf object is a Scala `object`, so declaring one is not
enough: its declarations only take effect once something touches it. Components need a hook that
Gluten invokes early enough for the backend channel — i.e. before any backend's driver/executor
startup, where native backend init happens. This is what makes the mechanism usable from
out-of-tree code, and it also makes declarations naturally modular: a backend's declarations only
exist when that backend is loaded, so Velox keys never leak into a ClickHouse deployment.

Prefix-based rules (`spark.gluten.sql.columnar.backend.`, `spark.hadoop.fs.s3a.`, …) are
out of scope — they cover open-ended key families and are pattern rules rather than enumerable
declarations.

#### Acceptance criteria

- No hard-coded native conf key list, default-value list, or per-key special case remains in the
selection methods.
- The delivered key/value set is unchanged for every existing conf, except where a mismatch is
deliberately corrected — each such difference stated explicitly.
- A conf's native delivery is readable from its definition alone.
- An out-of-tree backend or component can declare native confs without modifying Gluten common
code, verified from outside the `org.apache.gluten` package.
- The declared mutability of every native-passed conf matches where native consumes it.

This description was written with the assistance of generative AI tooling (Claude Code).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.