apache / apache/dolphinscheduler
[DSIP-22][TriggerPlugin] Introduce Trigger Plugin
- Dominant language
- Java
- Stars
- 14.5k
- Forks
- 5.1k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 28
Description
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar feature requirement.
### Description
After seeing all local & large distributed job-scheduling framework,
I found that we may leverage our excellent plugin framwork design and consider extending a key process in job-scheduling ---- **Trigger**, which allows user to defines their custom trigger in distributed runtime environment.
Apparently **quartz** has considered this situation, which is a part of master scheduler, but not extend into distributed executing semantics.
seeing https://www.quartz-scheduler.org/documentation/quartz-2.1.7/tutorials/tutorial-lesson-04.html
In distributed job-scheduling environment, **we could aligned all trigger into a event trigger.**
#### Use case
- **[Use case 1: TimerTrigger]**
This is common case. which is currently widely supported by dophinscheduler and all other job-scheduling framework, **which event is signal event from timer.**
- **[Use case 2: MQTrigger]**
for example, after receive a message with some pattern, we should start a new workflow. **which event is message from MQ.**
- **[Use case 3: HTTPTrigger/TCPTrigger/SMTPTrigger]**
very popular. after we make a HTTP call with dophinscheduler-api, we should start a new workflow. even we can trigger pipeline after we receive an email from external 3p team.
**which event is network call.**
- **[Use case 4: Combine Streaming & Batch Job]**
In common scenarios, we often use streaming to do some near-real-time OLAP job, which requires batch job for backfill supplyment in case some unexpected error happens in streaming.
**which event could be any condition with streaming elements.**
Benefits from the best plugin in dolphinscheduler.
We should consider to add trigger **SPI plugin loaded by master server.**
Overview:
```
┌─────────────────────┬────────────────────┬────────────────────┬────────────────────────┬───────────────────────────────┐
│ │ │ │ │ │
│ UI │ API │ DB │ Registry │ Master │
│ │ │ │ │ │
├─────────────────────┼────────────────────┼────────────────────┼────────────────────────┼───────────────────────────────┤
│ │ │ │ │ │
│ ┌─────────────┐ │ ┌─────────────┐ │ ┌─────────────┐ │ │ ┌─────────────────────────┐ │
│ │ │ │ │ │ │ │ │ │ │ │ │ │
│ │ User ├───┼──►│ Create ├──┼──►│ Trigger ├──┼──────────Pull──────────┼──► TriggerTaskThreadPool │ │
│ │ │ │ │ Trigger │ │ │ │ │ │ │ │ │
│ └─────┬───────┘ │ └─────────────┘ │ └─────────────┘ │ │ └───────────┬─────────────┘ │
│ │ │ │ │ │ │ │
│ │ │ ┌─────────────┐ │ ┌─────────────┐ │ │ │ │
│ │ │ │ │ │ │ │ │ │ │ │
│ └─Push──────┼──►│ Request ├──┼──►│ Schedule ◄──┼────────────────────────┼──────────────┘ │
│ │ │ │ │ │ │ │ │ │
│ │ └─────────────┘ │ └─────┬───────┘ │ │ │
│ │ │ │ │ │ │
│ │ │ │ │ │ ┌────────────────────────┐ │
│ │ │ │ │ │ │ │ │
│ │ │ └──────────┼────────────────────────┼───► SchedulerApi │ │
│ │ │ │ │ │ │ │
│ │ │ │ │ └───────────┬────────────┘ │
│ │ │ │ │ │ │
│ │ │ ┌─────────────┐ │ │ │ │
│ │ │ │ │ │ │ │ │
│ │ │ │ Command ◄──┼────────────────────────┼───────────────┘ │
│ │ │ │ │ │ │ │
│ │ │ └─────┬───────┘ │ │ │
│ │ │ │ │ │ │
│ │ │ │ │ │ │
│ │ │ │ │ │ ┌─────────────────────────┐ │
│ │ │ │ │ │ │ │ │
│ │ │ └──────────┼────────────────────────┼─►MasterSchedulerBootStrap │ │
│ │ │ │ │ └─────────────────────────┘ │
│ │ │ │ │ │
│ │ │ │ │ │
│ │ │ │ │ │
│ │ │ │ │ │
│ │ │ │ │ │
│ │ │ │ │ │
│ │ │ │ │ │
│ │ │ │ │ │
│ │ │ │ │ │
│ │ │ │ │ │
└─────────────────────┴────────────────────┴────────────────────┴────────────────────────┴───────────────────────────────┘
```
- All trigger should trigger by **Event**
- **Event Delivery** should based either **Push** or **Pull** mode.
- Trigger only at the start on the workflow for easy implementation.
### Detail Design
#### Controller Layer
- TriggerController CURD Trigger
#### Compatibility
- Scheduler has been refactor into scheduler-api partially.
- Relationship of trigger spi & scheduler api
#### DB Layer
- Create new table `t_ds_trigger`
- Leverage current table `t_ds_schedules`
- Use SimpleTrigger to execute immediately, so that we can update db accordingly in quartz scheduler implementation.
### Related issues
[[DSIP-16][Task] Support stream task](https://github.com/apache/dolphinscheduler/issues/11352)
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
Assessment
This issue has not been assessed yet.