dandi / dandi/backups2datalad

Use "batched" version of `git rm`

Open
#62 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
0
Forks
1
Avg merge
2d 16h
Merged PRs (30d)
5

Description

ATM, even though not downloading any context, it takes awhile to run some updates. Eg. is taking almost 20h for an update for 001169, which `13578 files changed, 13578 insertions(+)` .

I think part of the issue is that we are issuing an individual ` git -c receive.autogc=0 -c gc.auto=0 rm -f --ignore-unmatch --` which would need to read/modify/save git index which is quite heavy. there is no `git rm --stdin` but there is `git update-index --remove-force --stdin` so we could in parallel to it run `rm` to remove the file and then feed that path to `git update-index --remove-force --stdin`. I am not yet 100% certain it would provide for the ultimate speed up, as I think groupping of individual operations might provide better performance, but it might be worth to check. We do use `--batch` mode for many annex commands, but seems so far do not use any `--stdin` of `git`.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.