CCExtractor / CCExtractor/ccsync
Persistent Job Queue & Maintenance Worker for backend
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 37
- Forks
- 76
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 2
Description
**Is your feature request related to a problem? Please describe.**
The current job queue is volatile. If the backend container crashes or restarts, all queued tasks are lost forever. This needs to be persistent, so that occasional backend downtimes do not affect the Jobs. Additionally, as we implement persistence, we need a way to manage the lifecycle of the "Success" and "Failure" logs stored in the database to prevent unbounded disk usage.
**Describe the solution you'd like**
1. Implement a disk-backed queue using `bbolt` (or something similar, open to suggestions) and an internal Go-based cron scheduler for maintenance.
- The Queue will work as a database that will store all Jobs from the JobStore.
2. Persistence Layer (bbolt)
- Store the database file at /app/data/queue.db to use the existing Docker volume.
- Job States: Implement buckets or tables for Pending, In-Progress (doubt if it should exist, perhaps Pending, Completed, and Failed should suffice for our use case), and Completed statuses.
- Reliability: On startup, a NewJobQueue function must scan the Pending and In-Progress buckets and re-insert those jobs into the channel. Upon completion, mark them as completed.
3. Scheduled Maintenance (Cron)
- Integrate a package like robfig/cron/v3 to handle periodic tasks without external dependencies.
- Cleanup Job: Runs every 24 hours (or at midnight). Add that CRON parameter as an optional variable in .env file. Users might want to customise, but having a default value is nice, as well.
- Logic: Deletes all entries from the Completed or Failed buckets that are older than 7 days.
**Additional context**
Need to take into account the amount of storage that database will cost. And hence optimise it further. Documentation, related to the changes, related to env vars, and cron job along with the Database need to be added, to the Backend README. The current functionality should not be affected/broken.
- The issue is open to all for their versions of implementations.
Links:
https://github.com/etcd-io/bbolt
https://github.com/robfig/cron
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the Go JobStore and the NewJobQueue entry point, then review how the backend uses the Docker volume at /app/data and environment variables in .env. Define persistence for pending, in-progress, completed, and failed jobs, scheduled cleanup, and recovery after startup; update the Backend README with the database, cron, and environment-variable behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go
- Domain
- backend, databases, documentation
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100