CCExtractor / CCExtractor/ccsync

Persistent Job Queue & Maintenance Worker for backend

Open
#367 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

backend complex enhancement
Dominant language
TypeScript
Stars
37
Forks
76
Avg merge
3d 7h
Merged PRs (30d)
2

Description

**Is your feature request related to a problem? Please describe.**
The current job queue is volatile. If the backend container crashes or restarts, all queued tasks are lost forever. This needs to be persistent, so that occasional backend downtimes do not affect the Jobs. Additionally, as we implement persistence, we need a way to manage the lifecycle of the "Success" and "Failure" logs stored in the database to prevent unbounded disk usage.

**Describe the solution you'd like**
1. Implement a disk-backed queue using `bbolt` (or something similar, open to suggestions) and an internal Go-based cron scheduler for maintenance.
- The Queue will work as a database that will store all Jobs from the JobStore.

2. Persistence Layer (bbolt)
- Store the database file at /app/data/queue.db to use the existing Docker volume.
- Job States: Implement buckets or tables for Pending, In-Progress (doubt if it should exist, perhaps Pending, Completed, and Failed should suffice for our use case), and Completed statuses.
- Reliability: On startup, a NewJobQueue function must scan the Pending and In-Progress buckets and re-insert those jobs into the channel. Upon completion, mark them as completed.

3. Scheduled Maintenance (Cron)
- Integrate a package like robfig/cron/v3 to handle periodic tasks without external dependencies.
- Cleanup Job: Runs every 24 hours (or at midnight). Add that CRON parameter as an optional variable in .env file. Users might want to customise, but having a default value is nice, as well.
- Logic: Deletes all entries from the Completed or Failed buckets that are older than 7 days.

**Additional context**
Need to take into account the amount of storage that database will cost. And hence optimise it further. Documentation, related to the changes, related to env vars, and cron job along with the Database need to be added, to the Backend README. The current functionality should not be affected/broken.

- The issue is open to all for their versions of implementations.

Links:
https://github.com/etcd-io/bbolt
https://github.com/robfig/cron

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the Go JobStore and the NewJobQueue entry point, then review how the backend uses the Docker volume at /app/data and environment variables in .env. Define persistence for pending, in-progress, completed, and failed jobs, scheduled cleanup, and recovery after startup; update the Backend README with the database, cron, and environment-variable behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
backend, databases, documentation
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.