awslabs / awslabs/git-remote-s3

Unable to Push to Standard LFS Remote and S3 Remote Together

Open
#32 8 comments 0 reactions 1 assignee Claimed by @massi-ang View on GitHub
question
Dominant language
Python
Stars
823
Forks
42
Avg merge
8h 23m
Merged PRs (30d)
2

Description

I'm trying to use a git repository with multiple remotes, one to an internal GitFarm repository and another to an S3 bucket based repository. The repository has several large Git LFS artifacts that I want to publish to both GitFarm and S3. Essentially, I want the S3 bucket to act as a replica.

When I set up the new S3 remote, the LFS settings are not scoped to the S3 remote so I am not longer able to push to the GitFarm LFS.

### Before

```shell
❯ git remote -v
origin ssh://git.amazon.com/pkg/TK (fetch)
origin ssh://git.amazon.com/pkg/TK (push)

❯ git status
On branch main
Your branch is up to date with 'origin/main'.

nothing to commit, working tree clean

❯ git pull
Already up to date.

❯ git push
Everything up-to-date
```

### After adding S3 Remote unable to push to GitFarm

```shell
git remote add s3 s3://TK/TK
git config --add lfs.customtransfer.git-lfs-s3.path git-lfs-s3
git config --add lfs.standalonetransferagent git-lfs-s3
```

```shell
❯ git remote -v
origin ssh://git.amazon.com/pkg/TK (fetch)
origin ssh://git.amazon.com/pkg/TK (push)
s3 s3://TK/TK (fetch)
s3 s3://TK/TK (push)
```

```
❯ git push s3 main
Everything up-to-date

❯ git pull s3 main
From s3://TK/TK
* branch main -> FETCH_HEAD
Already up to date.
```

```shell
❯ cp output.parquet output2.parquet
❯ git add output2.parquet
❯ git commit -m "test add parquet file"
```

```shell
❯ git push origin main
error initializing custom adapter "git-lfs-s3" worker 0: [32] s3 uri ssh://git.amazon.com/pkg/TK is invalid
error: failed to push some refs to 'ssh://git.amazon.com/pkg/TK'
```

As you can see, I am unable to push to GitFarm now because it is trying to use the `git-lfs-s3`. I scoured the documentation and I could not find a way to configure `git-lfs-s3` only for the `s3` remote.

Is it possible to make `git-lfs-s3` act as a pass through if it is trying to push to a non-s3 uri?

### Git Config

Here is my full git config for reference:

```
❯ cat .git/config
[core]
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
[remote "origin"]
url = ssh://git.amazon.com/pkg/TK
fetch = +refs/heads/*:refs/remotes/origin/*
[branch "main"]
remote = origin
merge = refs/heads/main
[lfs]
repositoryformatversion = 0
standalonetransferagent = git-lfs-s3
[remote "s3"]
url = s3://TK/TK
fetch = +refs/heads/*:refs/remotes/s3/*
[lfs "customtransfer.git-lfs-s3"]
path = git-lfs-s3
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.