apache / apache/dolphinscheduler

[DSIP-91][Api] Workflow debugging and release optimization

Open
#17,252 21 comments 0 reactions 0 assignees View on GitHub
discussion DSIP
Dominant language
Java
Stars
14.5k
Forks
5.1k
Avg merge
1d 21h
Merged PRs (30d)
29

Description

### Search before asking

- [x] I had searched in the [DSIP](https://github.com/apache/dolphinscheduler/issues/14102) and found no similar DSIP.

### Motivation

#### Current Workflow Issues
##### Operational Process:

1. Development Process: For development workflows, the following steps must be executed: 1. Return to the workflow list -> 2. Deploy the workflow -> 3. Run the workflow -> 4. Check the runtime logs.
2. Modification Process: Adjusting workflow content is more complex: 1. Return to the workflow list -> 2. Undeploy the workflow -> 3. Enter workflow editing -> 4. Return to the workflow -> 5. Deploy the workflow -> 6. Run the workflow -> 7. Check the runtime logs.

##### Technical/Design Factors:

1. The workflow lacks a development state. While this simplifies logic and reduces bugs, it shifts complexity to the user.
2. The design prioritizes production stability, requiring users to be more cautious when making adjustments.

#### Desired:
Operational Process Optimization: Introduce a dedicated development workflow, separate from the production workflow (which can only be deployed from the development environment and cannot be created directly).

1. Development Workflow Process: Only requires: 1. Modify workflow content -> 2. Run the workflow -> 3. Check the runtime logs.
2. Deployment Process: Publish a development workflow to the production workflow (overwriting if previously deployed).

##### Technical/Design Optimization:

1. Development and production workflows are stored in separate tables with completely independent data.
2. Development workflows lack a deployment state, allowing debugging. Running workflows and task instances are isolated from production.
3. During deployment, the current version of the workflow is copied to the production workflow record, with an association logged (for subsequent overwrites).

### Design Detail

1. The workflow adds a label field to mark more information, such as whether it is a development or production workflow, which can also be customized

Image

2. The workflow operation adds an overwrite (replace) operation: click and select the workflow to overwrite the non-online state (can be across projects),It supports the comparison of differences between two nodes, the coverage is for the entire workflow.

Image

3. The workflow supports label filtering

Image

### Compatibility, Deprecation, and Migration Plan

There is a compatibility issue. This table is needed. Data migration needs to be considered. Complement processing is required in init-job.

### Test Plan

Old version workflow runs stock data upgrade verification
Functional testing
UT and E2E modification test

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Start with the compatibility and migration requirements, especially the init-job processing mentioned in the issue. Map the separate development and production workflow data, overwrite operation, labels, and isolated runs before reviewing the planned UT, E2E modification, functional, and stock-data upgrade tests. Done means the workflow debugging, deployment, migration, and compatibility behaviors are covered.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.