apache / apache/dolphinscheduler
[DSIP-109][Storage] Add Git repository as a resource storage backend
- Dominant language
- Java
- Stars
- 14.5k
- Forks
- 5.1k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 29
Description
### Search before asking
- [x] I had searched in the [DSIP](https://github.com/apache/dolphinscheduler/issues/14102) and found no similar DSIP.
### Motivation
DolphinScheduler currently supports LOCAL, HDFS, S3, OSS, GCS, ABS, OBS, COS as resource storage backends. These are all object/file storage systems. In many scenarios, users want to store and version-control workflow resources (shell scripts, SQL files, Python files, UDFs) directly in a Git repository, gaining:
1. Version history — every resource change is a Git commit with a diffable audit trail
2. GitOps workflow — resources managed through pull requests / code review
3. Git-native CI/CD integration — resources can trigger CI pipelines natively
This DSIP proposes adding a new storage plugin module dolphinscheduler-storage-git that implements the existing StorageOperator SPI, supporting GitHub, Gitea, and GitLab as storage backends via their REST Contents/Files APIs.
### Design Detail
Implements StorageOperator. Maps DS file operations (create/exists/delete/copy/upload/download/list) to Git API calls via REST.
github:
https://docs.github.com/en/rest/authentication/authenticating-to-the-rest-api?apiVersion=2026-03-10
https://docs.github.com/en/rest/repos/contents?apiVersion=2026-03-10
gitlab:
https://docs.gitlab.com/api/rest/#authentication
https://docs.gitlab.com/api/repository_files/
gitea:
https://docs.gitea.com/development/api-usage
https://docs.gitea.com/api/1.25/#tag/repository/operation/repoMigrate
I can submit a PR!
### Compatibility, Deprecation, and Migration Plan
This is a purely additive feature
### Test Plan
GitLab/gitea can use container test.
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
Research direction
Start by reading the existing StorageOperator SPI and storage plugin structure. Review the linked GitHub, GitLab, and Gitea REST API documentation, then use the proposed container-based tests for GitLab and Gitea. Done means the plugin maps create, exists, delete, copy, upload, download, and list operations across the supported Git backends.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- git, github, gitlab, java
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100