apache / apache/dolphinscheduler

[DSIP-22][TriggerPlugin] Introduce Trigger Plugin

Open
#15,330 12 comments 8 reactions 1 assignee Claimed by @pegasas View on GitHub
DSIP feature
Dominant language
Java
Stars
14.5k
Forks
5.1k
Avg merge
1d 21h
Merged PRs (30d)
28

Description

### Search before asking

- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar feature requirement.

### Description

After seeing all local & large distributed job-scheduling framework,
I found that we may leverage our excellent plugin framwork design and consider extending a key process in job-scheduling ---- **Trigger**, which allows user to defines their custom trigger in distributed runtime environment.

Apparently **quartz** has considered this situation, which is a part of master scheduler, but not extend into distributed executing semantics.
seeing https://www.quartz-scheduler.org/documentation/quartz-2.1.7/tutorials/tutorial-lesson-04.html

In distributed job-scheduling environment, **we could aligned all trigger into a event trigger.**

#### Use case

- **[Use case 1: TimerTrigger]**
This is common case. which is currently widely supported by dophinscheduler and all other job-scheduling framework, **which event is signal event from timer.**
- **[Use case 2: MQTrigger]**
for example, after receive a message with some pattern, we should start a new workflow. **which event is message from MQ.**
- **[Use case 3: HTTPTrigger/TCPTrigger/SMTPTrigger]**
very popular. after we make a HTTP call with dophinscheduler-api, we should start a new workflow. even we can trigger pipeline after we receive an email from external 3p team.
**which event is network call.**
- **[Use case 4: Combine Streaming & Batch Job]**
In common scenarios, we often use streaming to do some near-real-time OLAP job, which requires batch job for backfill supplyment in case some unexpected error happens in streaming.
**which event could be any condition with streaming elements.**

Benefits from the best plugin in dolphinscheduler.
We should consider to add trigger **SPI plugin loaded by master server.**

Overview:

```
┌─────────────────────┬────────────────────┬────────────────────┬────────────────────────┬───────────────────────────────┐
│ │ │ │ │ │
│ UI │ API │ DB │ Registry │ Master │
│ │ │ │ │ │
├─────────────────────┼────────────────────┼────────────────────┼────────────────────────┼───────────────────────────────┤
│ │ │ │ │ │
│ ┌─────────────┐ │ ┌─────────────┐ │ ┌─────────────┐ │ │ ┌─────────────────────────┐ │
│ │ │ │ │ │ │ │ │ │ │ │ │ │
│ │ User ├───┼──►│ Create ├──┼──►│ Trigger ├──┼──────────Pull──────────┼──► TriggerTaskThreadPool │ │
│ │ │ │ │ Trigger │ │ │ │ │ │ │ │ │
│ └─────┬───────┘ │ └─────────────┘ │ └─────────────┘ │ │ └───────────┬─────────────┘ │
│ │ │ │ │ │ │ │
│ │ │ ┌─────────────┐ │ ┌─────────────┐ │ │ │ │
│ │ │ │ │ │ │ │ │ │ │ │
│ └─Push──────┼──►│ Request ├──┼──►│ Schedule ◄──┼────────────────────────┼──────────────┘ │
│ │ │ │ │ │ │ │ │ │
│ │ └─────────────┘ │ └─────┬───────┘ │ │ │
│ │ │ │ │ │ │
│ │ │ │ │ │ ┌────────────────────────┐ │
│ │ │ │ │ │ │ │ │
│ │ │ └──────────┼────────────────────────┼───► SchedulerApi │ │
│ │ │ │ │ │ │ │
│ │ │ │ │ └───────────┬────────────┘ │
│ │ │ │ │ │ │
│ │ │ ┌─────────────┐ │ │ │ │
│ │ │ │ │ │ │ │ │
│ │ │ │ Command ◄──┼────────────────────────┼───────────────┘ │
│ │ │ │ │ │ │ │
│ │ │ └─────┬───────┘ │ │ │
│ │ │ │ │ │ │
│ │ │ │ │ │ │
│ │ │ │ │ │ ┌─────────────────────────┐ │
│ │ │ │ │ │ │ │ │
│ │ │ └──────────┼────────────────────────┼─►MasterSchedulerBootStrap │ │
│ │ │ │ │ └─────────────────────────┘ │
│ │ │ │ │ │
│ │ │ │ │ │
│ │ │ │ │ │
│ │ │ │ │ │
│ │ │ │ │ │
│ │ │ │ │ │
│ │ │ │ │ │
│ │ │ │ │ │
│ │ │ │ │ │
│ │ │ │ │ │
└─────────────────────┴────────────────────┴────────────────────┴────────────────────────┴───────────────────────────────┘
```

- All trigger should trigger by **Event**
- **Event Delivery** should based either **Push** or **Pull** mode.
- Trigger only at the start on the workflow for easy implementation.

### Detail Design

#### Controller Layer

- TriggerController CURD Trigger

#### Compatibility

- Scheduler has been refactor into scheduler-api partially.
- Relationship of trigger spi & scheduler api

#### DB Layer

- Create new table `t_ds_trigger`
- Leverage current table `t_ds_schedules`
- Use SimpleTrigger to execute immediately, so that we can update db accordingly in quartz scheduler implementation.

### Related issues

[[DSIP-16][Task] Support stream task](https://github.com/apache/dolphinscheduler/issues/11352)

### Are you willing to submit a PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.