Clean up S3 Bucket - and other places
Open
Nobody has claimed this yet.
deployment
- Dominant language
- C#
- Stars
- 3
- Forks
- 2
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 3
Description
There should be an agent that periodically runs that clears data that is no longer used in the S3 bucket and in the main /var/lib folders. It should include:
- Getting rid of files with no Mongo DB entry
- Getting rid of engines with no Mongo DB entry in /var/lib/engines
- Getting rid of S3 bucket experiments with no Mongo DB entry, or that have been completed successfully and finished.
Note that:
- We need to ensure that the Mongo DB entry will be created before the files are created, otherwise, we could have race conditions and delete files before they are entered into the database. Same with pulling all files from the S3 bucket before calling the job "ready do delete" or some such thing.
- This job should run periodically (once a day?) to clean up files. It needs to clear files even if there was an unexpected shutdown.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the service entry point and configuration for the S3 bucket, Mongo DB records, and /var/lib/engines. Define the cleanup job's ownership and schedule, then verify that unreferenced files and completed experiments are removed safely after database entries exist and that cleanup can recover from an unexpected shutdown.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, csharp, mongodb
- Domain
- backend, cloud, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100