[CORE][SPARK-4.0] Only require one config spark.plugins to enable gluten
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Description
For now, there are at least three required configs to enable gluten:
- spark.plugins
- spark.memory.offHeap.enabled
- spark.shuffle.manager
Less required configs is always better. Fortunately, We can set `spark.memory.offHeap.enabled` and `spark.shuffle.manager` inside `GlutenDriverPlugin` since Spark 4.0.0, so only `spark.plugins` is required.
- spark.memory.offHeap.enabled: https://github.com/apache/spark/pull/45052
- spark.shuffle.manager: https://github.com/apache/spark/pull/43627
Contributor guide
Research direction
Start by reading the Spark 4.0 integration around GlutenDriverPlugin and the existing handling of spark.memory.offHeap.enabled and spark.shuffle.manager. Verify the plugin can set both options when only spark.plugins is configured, and check the relevant integration or configuration tests if present. Done means Gluten enables with only spark.plugins under Spark 4.0.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100