apache / apache/dolphinscheduler

[DSIP-79][Task] Add Datavines task to better support data quality

Open
#16,113 10 comments 2 reactions 1 assignee Claimed by @zixi0825 View on GitHub
DSIP help wanted
Dominant language
Java
Stars
14.5k
Forks
5.1k
Avg merge
1d 21h
Merged PRs (30d)
28

Description

### Search before asking

- [X] I had searched in the [DSIP](https://github.com/apache/dolphinscheduler/issues/14102) and found no similar DSIP.

### Motivation

DataVines is an easy-to-use data quality service platform that supports multiple metric.
https://github.com/datavane/datavines

- Datavines supports executing multiple metrics in one job.
- Datavines supports execution status dashboard and data quality report.
- Datavines supports plug-in extensions for components such as metric, data sources, error data storage, and execution engines.
- Jdbc engines can be used to execute data quality tasks instead of solely relying on Spark engines.

### Design Detail

#### Sript mode
1. config data quality job in datavines
![image](https://github.com/apache/dolphinscheduler/assets/58384836/8ff2cf5e-4d09-4199-96bb-7a9326fcf8f2)

2. get the job config scipt file

3. Add datavines job node in workflow, and configure the script
![image](https://github.com/apache/dolphinscheduler/assets/58384836/888a9950-e4c2-49e7-ab2f-f9eb38aa5c4c)

#### API Mode
1. config data quality job in datavines
![image](https://github.com/apache/dolphinscheduler/assets/58384836/8ff2cf5e-4d09-4199-96bb-7a9326fcf8f2)

2. get the jobId

3. Add datavines job node in workflow, and configure the datavines api address and jobId

### Compatibility, Deprecation, and Migration Plan

_No response_

### Test Plan

_No response_

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.