apache / apache/dolphinscheduler

[DSIP-109][Storage] Add Git repository as a resource storage backend

Open
#18,350 13 comments 0 reactions 0 assignees View on GitHub
discussion DSIP
Dominant language
Java
Stars
14.5k
Forks
5.1k
Avg merge
1d 21h
Merged PRs (30d)
29

Description

### Search before asking

- [x] I had searched in the [DSIP](https://github.com/apache/dolphinscheduler/issues/14102) and found no similar DSIP.

### Motivation

DolphinScheduler currently supports LOCAL, HDFS, S3, OSS, GCS, ABS, OBS, COS as resource storage backends. These are all object/file storage systems. In many scenarios, users want to store and version-control workflow resources (shell scripts, SQL files, Python files, UDFs) directly in a Git repository, gaining:

1. Version history — every resource change is a Git commit with a diffable audit trail
2. GitOps workflow — resources managed through pull requests / code review
3. Git-native CI/CD integration — resources can trigger CI pipelines natively

This DSIP proposes adding a new storage plugin module dolphinscheduler-storage-git that implements the existing StorageOperator SPI, supporting GitHub, Gitea, and GitLab as storage backends via their REST Contents/Files APIs.

### Design Detail

Implements StorageOperator. Maps DS file operations (create/exists/delete/copy/upload/download/list) to Git API calls via REST.

github:
https://docs.github.com/en/rest/authentication/authenticating-to-the-rest-api?apiVersion=2026-03-10
https://docs.github.com/en/rest/repos/contents?apiVersion=2026-03-10
gitlab:
https://docs.gitlab.com/api/rest/#authentication
https://docs.gitlab.com/api/repository_files/
gitea:
https://docs.gitea.com/development/api-usage
https://docs.gitea.com/api/1.25/#tag/repository/operation/repoMigrate

I can submit a PR!

### Compatibility, Deprecation, and Migration Plan

This is a purely additive feature

### Test Plan

GitLab/gitea can use container test.

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Start by reading the existing StorageOperator SPI and storage plugin structure. Review the linked GitHub, GitLab, and Gitea REST API documentation, then use the proposed container-based tests for GitLab and Gitea. Done means the plugin maps create, exists, delete, copy, upload, download, and list operations across the supported Git backends.

Written by the indexing model from the issue text.

Assessment

Tech stack
git, github, gitlab, java
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.