apache / apache/dolphinscheduler

[DSIP-106][WorkerGroup] Persist All Worker Groups in the Database

Open
#17,890 3 comments 0 reactions 1 assignee Claimed by @ruanwenjun View on GitHub
backend discussion DSIP improvement
Dominant language
Java
Stars
14.5k
Forks
5.1k
Avg merge
1d 21h
Merged PRs (30d)
29

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar feature requirement.

### Description

The worker group represents a collection of workers, and every task must be associated with a worker group. However, the current worker group design has several usability and management issues.

#### Existing Problems

##### Worker group names are tightly coupled with permissions and tasks.
Worker group names are directly referenced in both permission control and task configuration. Once a worker group name is changed, existing permissions and tasks become invalid.

##### The default worker group is difficult to manage.
The default worker group implicitly contains all workers, and all tasks are allowed to use it. This makes resource isolation and governance nearly impossible.

##### A worker can only be dynamically bound to a single worker group via configuration.
This limits flexibility in resource organization and prevents a worker from serving multiple logical groups.

#### Proposal

This DSIP proposes improvements to the worker group mechanism:

##### All worker groups must have metadata persisted in the database.
Worker groups should become first-class entities with unified management, rather than being partially configuration-driven.

##### Workers can still dynamically register to worker groups via configuration, but the worker group must be pre-defined in the API server.
This ensures consistency and prevents accidental or inconsistent group creation.

##### A worker should be allowed to dynamically register to multiple worker groups.
This improves flexibility in resource sharing and logical grouping.

##### The default worker group should behave like a regular worker group and be empty by default.
Workers must explicitly join it, enabling proper governance and isolation.

### Are you willing to submit a PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.