[Umbrella] Spark Configuration Validation
- Dominant language
- Scala
- Stars
- 2.4k
- Forks
- 1k
- PR merge metrics
- No merged PRs in 30d
Description
# Describe the proposal
When a SparkConf is invalid for launching our engines, it is better to fail fast instead of waiting for Spark to check while engine bootstrapping.
## Motivation
1. Save time to create an invalid engine
2. Avoid possible hang
3. better error message
Here is an example I did for Spark to validate its own configs when talking to Kubernetes. https://github.com/apache/spark/pull/32610
# Task list
Generally speaking, we can have 1) a tool in `kyuubi-ctl` to check server/engine conf in configuration files are valid, 2) The logic of the tool can also be called when engine bootstrapping for the configurations from the client side
- [x] #833
- [x] #1018
Contributor guide
Research direction
Start by reviewing the kyuubi-ctl configuration-checking tool and the engine bootstrapping path described in the issue. Compare the completed tasks in #833 and #1018 to determine what remains; the proposal is done when server/engine configuration files and client-side configurations are validated before engine startup.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- backend, cli, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100