dandi / dandi/dandisets

.gc within zarr_sync is not "in effect"?

Open
#365 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
15
Forks
2
PR merge metrics
No merged PRs in 30d

Description

while assessing what consumes space I spotted that some zarrs have huge number of loose objects, which seems to be packed or cleaned up with `git gc` call. But it seems that we do call `git gc` after syncing zarr: https://github.com/dandi/dandisets/blob/306e1ae3732e8f44fbf79ad4b44e9bb7bf07dc2b/tools/backups2datalad/zarr.py#L577
so it is not clear why we still have the case (may be failed syncs which never got `git gc` ran, only me doing `git reset --hard`?). I am still collecting stats across all zarrs under `/tmp/dandizarrs-inodes.out` but here is an example of one and effect of running `git gc` now:

```shell
dandi@drogon:/mnt/backup/dandi/dandizarrs/0cf3c734-4bb2-47d1-8e6a-528a3739af01$ du -scm --inodes .git/objects/; git gc; du -scm --inodes .git/objects/
232704 .git/objects/
232704 total
Enumerating objects: 541506, done.
Counting objects: 100% (541506/541506), done.
Delta compression using up to 4 threads
Compressing objects: 100% (469070/469070), done.
Writing objects: 100% (541506/541506), done.
Total 541506 (delta 224707), reused 156961 (delta 72419), pack-reused 0
Removing duplicate objects: 100% (256/256), done.
7 .git/objects/
7 total
```

so we went from 230k to 7 inodes in .git/objects.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.