apache / apache/fluss

[Spark] Add set_cluster_configs and reset_cluster_configs procedures for Spark connector

Open
#3,203 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Motivation

Currently, the Spark connector only supports the `get_cluster_configs` procedure for cluster configuration management. However, the Flink connector already provides a complete set of cluster configuration management procedures, including `get_cluster_configs`, `set_cluster_configs`, and `reset_cluster_configs`.

This inconsistency means that Spark users cannot dynamically modify or reset cluster configurations through SQL, which limits the usability of the Spark connector. Users who need to change dynamic cluster configurations (e.g., `kv.rocksdb.shared-rate-limiter.bytes-per-sec`) have to switch to the Flink connector or use other tools, which is inconvenient.

We should align the Spark connector's procedure capabilities with the Flink connector by adding `set_cluster_configs` and `reset_cluster_configs` procedures.

### Solution

Add two new procedures to the Spark connector:

1. **`set_cluster_configs`** - Dynamically set cluster configuration values. Accepts an array of key-value pairs and applies them to the cluster.
2. **`reset_cluster_configs`** - Reset cluster configurations to their default values. Accepts an array of configuration keys to reset.

### Anything else?

_No response_

### Willingness to contribute

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the Spark connector's existing get_cluster_configs procedure and compare it with the Flink connector's set_cluster_configs and reset_cluster_configs procedures. Trace their SQL procedure registration and related tests, then add Spark coverage showing that configuration values can be set and reset through SQL.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark, sql
Domain
backend, data-engineering
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.