apache / apache/gluten

[CORE] Configuration enhancements

Open
#6,970 12 comments 4 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Description

Gluten's configuration utilities, especially `GlutenConfig.scala`, is getting messed up over time. To keep code clean and easy to maintain, will propose the below changes:

### Simplify and unify option key formats

1. Could simplify config key prefix from `spark.gluten.sql.columnar` tp `spark.gluten`.
2. Use camel case uniformly, e.g, `spark.gluten.sql.columnar.backend.velox.IOThreads` to `spark.gluten.sql.columnar.backend.velox.ioThreads`

### Move specific configurations to backend module

For better module-wise isolation, we should move configuration definitions specifically for backends to their own module. E.g., `spark.gluten.sql.columnar.backend.velox.window.type` should be defined in `backends-velox` than `gluten-core`.

### (Velox) Remove the list of configurations passing to native

In Velox backend, we currently have a list to decide whether a config option should be passed to native. Rather than using this way, we can just set a flag when define the config. Or just using Spark's built-in static / non-static config flag to distinguish.

### Backward compatibility

Since configuration keys may be modified during doing this, we should maintain a list of configuration keys for backward compatibility. E.g., `spark.gluten.sql.columnar.backend.velox.IOThreads` can be the alternative of `spark.gluten.backend.velox.ioThreads`. But we should raise a warning when the former is set by user as it's considered deprecated. After several development iterations, support of the old one will be removed completely.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing GlutenConfig.scala and the configuration handling in the backends-velox module. Trace how backend options are defined, how Velox decides which options reach native code, and how aliases or deprecation warnings could be represented. Done means the key formats are unified, backend-specific definitions are isolated, native forwarding no longer relies on the existing list, and old keys remain supported with warnings.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
backend
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.