apache / apache/dolphinscheduler

[Improvement][AliyunServerlessSparkTask] Some Improvement on Aliyun Serverless Spark Task Plugin

Open
#16,994 5 comments 1 reaction 1 assignee Claimed by @abzymeinsjtu View on GitHub
backend improvement priority:middle
Dominant language
Java
Stars
14.5k
Forks
5.1k
Avg merge
1d 21h
Merged PRs (30d)
28

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar feature requirement.

### Description

@EricGao888 has released a plugin earlier to implement batch job submission on Aliyun Serverless Spark. After a careful review, i think there are still something to be optimized. as listed below:

1. retry for remote api call is missing now, which may cause inconsistence state when some temporary issue occur like network failure
2. idempotent submitting is necessary especially for the same scheduled node in the same process instance
3. termination should be ensured cuz a single call on CancelJobRun Api can only turn a remote job into cancelling state, but not really canceled

### Are you willing to submit a PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.