apache / apache/opendal

new feature: Generic git service with LFS support

Open
#6,831 5 comments 1 reaction 0 assignees View on GitHub
enhancement releases-note/feat services/http
Dominant language
Rust
Stars
5.4k
Forks
825
Avg merge
1d 14m
Merged PRs (30d)
127

Description

### Feature Description

Currently there is no support for arbitrary git service over http with LFS support.

### Problem and Solution

### Problem:

We have a project that relies heavily on OpenDAL for remote data access, dealing primarily with AI data like weights and datasets. The HuggingFace service is awesome for this (as is s3 and others)! But we want to be able to support any git repository that may house the same sort of data, including internal repositories running in our self-hosted Gitlab instance.

We currently launch git in a subprocess to fetch and download ref history, then checkout the right commit, then clone LFS files - this feels clunky in a Rust application, and requires us to complete downloading the model before we can stream the contents to clients.

### Solution:

Using [gix](https://github.com/GitoxideLabs/gitoxide/) along with the OpenDAL http service, I was able to make a functioning prototype that can fetch the remote state of any remote repo at any ref or oid, pull the repository files, then go through the LFS pointers and start streaming them down with OpenDAL http service.

This fits our particular use case, but before I say good enough and call it a day, I wanted to know if this functionality might be of interest to the project maintainers here - and if so what is a good way to get this formally added as a feature request, and eventually contribute a crate feature? I wouldn't want to go through the effort of porting what I did thus far to OpenDAL's service APIs unless I had a path forward. Thanks!

### Additional Context

_No response_

### Are you willing to contribute to the development of this feature?

- [x] Yes, I am willing to contribute to the development of this feature.

Contributor guide

Open the contributing guide

Research direction

Review the existing OpenDAL service APIs and the prototype approach using gix with the OpenDAL HTTP service. Define how arbitrary Git repositories, refs or OIDs, Git LFS pointers, and streaming downloads should fit the service and crate-feature model; done means maintainers agree on the integration path and the service supports the stated workflow.

Written by the indexing model from the issue text.

Assessment

Tech stack
git, rust
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.