dandi / dandi/dandi-infrastructure

GC old inventory listings

Open
#198 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
HCL
Stars
0
Forks
11
Avg merge
2d 20h
Merged PRs (30d)
2

Description

As "discovered" in
- https://github.com/dandi/dandi-utils/issues/3

we might not really need historical records of inventory to achieve a "full backup" of S3. Inventory dumps themselves are quite large! I am still fetching (to facilitate analysis etc, but might stop doing that) and so far fetched 14TB. As such, it is a notable amount of storage . Here is how they grew through the years (per day)

```
(dandisets-2) dandi@drogon:/mnt/backup/dandi/dandiarchive-inventory$ code/print-manifest-summary dump/202*-01-01T*/manifest.json
dump/2020-01-01T00-00Z/manifest.json : 1 entries, 197K total size
dump/2021-01-01T00-00Z/manifest.json : 1 entries, 3.8M total size
dump/2022-01-01T00-00Z/manifest.json : 1 entries, 17M total size
dump/2023-01-01T01-00Z/manifest.json : 384 entries, 36G total size
dump/2024-01-01T01-00Z/manifest.json : 406 entries, 38G total size
```
and this year grew to 39G per day(!) which would amount 14TB per year just for the dumps (so I expect to fetch then 40TB... may be should interrupt and fetch specific days and their data only).

Mostly it is due to all the `zarr/`s. But it remains the case that we might want to prune some old inventory listings soonish. (attn @satra with whom we briefly discussed some bucket GCing to do)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.