[CORE] Configuration enhancements
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Description
Gluten's configuration utilities, especially `GlutenConfig.scala`, is getting messed up over time. To keep code clean and easy to maintain, will propose the below changes:
### Simplify and unify option key formats
1. Could simplify config key prefix from `spark.gluten.sql.columnar` tp `spark.gluten`.
2. Use camel case uniformly, e.g, `spark.gluten.sql.columnar.backend.velox.IOThreads` to `spark.gluten.sql.columnar.backend.velox.ioThreads`
### Move specific configurations to backend module
For better module-wise isolation, we should move configuration definitions specifically for backends to their own module. E.g., `spark.gluten.sql.columnar.backend.velox.window.type` should be defined in `backends-velox` than `gluten-core`.
### (Velox) Remove the list of configurations passing to native
In Velox backend, we currently have a list to decide whether a config option should be passed to native. Rather than using this way, we can just set a flag when define the config. Or just using Spark's built-in static / non-static config flag to distinguish.
### Backward compatibility
Since configuration keys may be modified during doing this, we should maintain a list of configuration keys for backward compatibility. E.g., `spark.gluten.sql.columnar.backend.velox.IOThreads` can be the alternative of `spark.gluten.backend.velox.ioThreads`. But we should raise a warning when the former is set by user as it's considered deprecated. After several development iterations, support of the old one will be removed completely.
Contributor guide
Research direction
Start by reviewing GlutenConfig.scala and the configuration handling in the backends-velox module. Trace how backend options are defined, how Velox decides which options reach native code, and how aliases or deprecation warnings could be represented. Done means the key formats are unified, backend-specific definitions are isolated, native forwarding no longer relies on the existing list, and old keys remain supported with warnings.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- backend
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100