Garbage collection [Meta issue]
- Dominant language
- Python
- Stars
- 26
- Forks
- 21
- Avg merge
- 4d 23h
- Merged PRs (30d)
- 15
Description
I am adding to migration target since we better iron it out before switching to dandi-api: bugs in GC can lead to data loss thus IMHO we first need make sure (as with extensive unit-testing and user-testing) that it works reliably before deploying it.
Initial design sketch on garbage collection is present within https://github.com/dandi/dandi-api/pull/150/files#diff-c96d4444d1714a52d5d08dd92d94919393a7db8ded038aa84f02ba1075d2c25eR37 but I think it is worth removing it from that PR and starting a new dedicated one.
I see following targets for GC
- uploads: not sure how if we could query S3 for some timestamp of the last activity for the upload urls
- blobs: any blob not linked to an asset is subject to GC
- GCing of a blob (blob_id) should also GC the actual key file in the store. We have 1-to-1 relationship ATM
- assets: any asset not linked to any (released or draft) version of a dandiset
- (edited) zarrs : like a blob, a zarr file which is not present in a dandiset it is linked to
Additional aspects:
- Might need DB-level locking unless we `touch` any blob or asset upon "being queried", since otherwise we might GC a blob in the middle of an asset being "minted" for an existing blob; very less likely but I guess could happen for assets in GC of an asset is triggered while we are "modifying" it and creating a new asset
- For any of those we might wan to allow for some "expiry" duration, so even if not "linked", but newer than e.g. a week (or 24h) -- keep around; If we add `touch` timestamping, this would allow to avoid locking (but would be costly for DB)
- Bucket is versioned (yeap -- on my insistence), so there might be need for a 2nd stage GCing picking up elderly/stale keys in S3, unless removing of a key would explicitly take care about removing it (not just adding DeleteMarker)
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.