dandi / dandi/dandi-archive

Garbage collection [Meta issue]

Open
#177 8 comments 0 reactions 1 assignee Claimed by @mvandenburgh View on GitHub
enhancement gc
Dominant language
Python
Stars
26
Forks
21
Avg merge
4d 23h
Merged PRs (30d)
15

Description

I am adding to migration target since we better iron it out before switching to dandi-api: bugs in GC can lead to data loss thus IMHO we first need make sure (as with extensive unit-testing and user-testing) that it works reliably before deploying it.

Initial design sketch on garbage collection is present within https://github.com/dandi/dandi-api/pull/150/files#diff-c96d4444d1714a52d5d08dd92d94919393a7db8ded038aa84f02ba1075d2c25eR37 but I think it is worth removing it from that PR and starting a new dedicated one.

I see following targets for GC

- uploads: not sure how if we could query S3 for some timestamp of the last activity for the upload urls
- blobs: any blob not linked to an asset is subject to GC
- GCing of a blob (blob_id) should also GC the actual key file in the store. We have 1-to-1 relationship ATM
- assets: any asset not linked to any (released or draft) version of a dandiset
- (edited) zarrs : like a blob, a zarr file which is not present in a dandiset it is linked to

Additional aspects:

- Might need DB-level locking unless we `touch` any blob or asset upon "being queried", since otherwise we might GC a blob in the middle of an asset being "minted" for an existing blob; very less likely but I guess could happen for assets in GC of an asset is triggered while we are "modifying" it and creating a new asset
- For any of those we might wan to allow for some "expiry" duration, so even if not "linked", but newer than e.g. a week (or 24h) -- keep around; If we add `touch` timestamping, this would allow to avoid locking (but would be costly for DB)
- Bucket is versioned (yeap -- on my insistence), so there might be need for a 2nd stage GCing picking up elderly/stale keys in S3, unless removing of a key would explicitly take care about removing it (not just adding DeleteMarker)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.