apache / apache/dolphinscheduler

[DSIP-100][Api Worker] Generic Third-party System API Connector for REST-based Task Scheduling

Open
#17,501 2 comments 0 reactions 1 assignee Claimed by @dill21yu View on GitHub
backend DSIP UI
Dominant language
Java
Stars
14.5k
Forks
5.1k
Avg merge
1d 21h
Merged PRs (30d)
29

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar feature requirement.

### Description

Propose a Generic Third-party System API Connector to enable DolphinScheduler to schedule tasks on any external system via REST APIs.

Currently, DS lacks a unified way to integrate with systems like SeaTunnel, Flink Session, or custom platforms that expose task control via RESTful interfaces. This feature would provide a configurable framework (managing endpoints for auth, task list, start, status, stop) instead of requiring custom plugins for each system, greatly expanding DS's scheduling capabilities.

### Use case

As a DS administrator or user,
I want to register an external system (e.g., company's customized SeaTunnel system) or other by providing its base URL, authentication, and specific API endpoints (to list, start, check status, and stop tasks),
So that I can easily select and schedule that system's tasks directly within a DolphinScheduler workflow.

The expected behavior is:
1、In the DS UI, I can add a new "Third-party System" config for our SeaTunnel server's API.
2、When building a workflow, I can choose the "Generic API Task" type, select the registered "SeaTunnel" system, and see a dropdown list of available jobs fetched from its API.
3、Upon selecting a job and running the workflow, DS automatically calls the start API, polls the status API until completion, and reflects the final status (success/failure) in DS.
4、I can also manually stop the task from DS, which triggers the stop API call to the external system.
This will allow us to use DS as a unified scheduler for many API-based tools without waiting for specific plugin development.

Image

Image

Image

Image

Image

### Related issues

_No response_

### Are you willing to submit a PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.