sillsdev / sillsdev/serval

Clean up S3 Bucket - and other places

Open
#251 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

deployment
Dominant language
C#
Stars
3
Forks
2
Avg merge
3d 1h
Merged PRs (30d)
3

Description

There should be an agent that periodically runs that clears data that is no longer used in the S3 bucket and in the main /var/lib folders. It should include:

  • Getting rid of files with no Mongo DB entry
  • Getting rid of engines with no Mongo DB entry in /var/lib/engines
  • Getting rid of S3 bucket experiments with no Mongo DB entry, or that have been completed successfully and finished.

Note that:

  • We need to ensure that the Mongo DB entry will be created before the files are created, otherwise, we could have race conditions and delete files before they are entered into the database. Same with pulling all files from the S3 bucket before calling the job "ready do delete" or some such thing.
  • This job should run periodically (once a day?) to clean up files. It needs to clear files even if there was an unexpected shutdown.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the service entry point and configuration for the S3 bucket, Mongo DB records, and /var/lib/engines. Define the cleanup job's ownership and schedule, then verify that unreferenced files and completed experiments are removed safely after database entries exist and that cleanup can recover from an unexpected shutdown.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, csharp, mongodb
Domain
backend, cloud, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.