apache / apache/kyuubi

[Umbrella] Spark Configuration Validation

Open
#832 1 comment 1 reaction 0 assignees View on GitHub
good first issue help wanted kind:feature
Dominant language
Scala
Stars
2.4k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

# Describe the proposal

When a SparkConf is invalid for launching our engines, it is better to fail fast instead of waiting for Spark to check while engine bootstrapping.

## Motivation

1. Save time to create an invalid engine
2. Avoid possible hang
3. better error message

Here is an example I did for Spark to validate its own configs when talking to Kubernetes. https://github.com/apache/spark/pull/32610

# Task list

Generally speaking, we can have 1) a tool in `kyuubi-ctl` to check server/engine conf in configuration files are valid, 2) The logic of the tool can also be called when engine bootstrapping for the configurations from the client side

- [x] #833
- [x] #1018

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the kyuubi-ctl configuration-checking tool and the engine bootstrapping path described in the issue. Compare the completed tasks in #833 and #1018 to determine what remains; the proposal is done when server/engine configuration files and client-side configurations are validated before engine startup.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
backend, cli, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.