CCExtractor / CCExtractor/ccsync
Persistent Job Queue & Maintenance Worker for backend
- 主要言語
- TypeScript
- スター
- 37
- フォーク
- 76
- 平均マージ
- 3日 7時間
- マージ済み PR(30日)
- 2
説明
**Is your feature request related to a problem? Please describe.**
The current job queue is volatile. If the backend container crashes or restarts, all queued tasks are lost forever. This needs to be persistent, so that occasional backend downtimes do not affect the Jobs. Additionally, as we implement persistence, we need a way to manage the lifecycle of the "Success" and "Failure" logs stored in the database to prevent unbounded disk usage.
**Describe the solution you'd like**
1. Implement a disk-backed queue using `bbolt` (or something similar, open to suggestions) and an internal Go-based cron scheduler for maintenance.
- The Queue will work as a database that will store all Jobs from the JobStore.
2. Persistence Layer (bbolt)
- Store the database file at /app/data/queue.db to use the existing Docker volume.
- Job States: Implement buckets or tables for Pending, In-Progress (doubt if it should exist, perhaps Pending, Completed, and Failed should suffice for our use case), and Completed statuses.
- Reliability: On startup, a NewJobQueue function must scan the Pending and In-Progress buckets and re-insert those jobs into the channel. Upon completion, mark them as completed.
3. Scheduled Maintenance (Cron)
- Integrate a package like robfig/cron/v3 to handle periodic tasks without external dependencies.
- Cleanup Job: Runs every 24 hours (or at midnight). Add that CRON parameter as an optional variable in .env file. Users might want to customise, but having a default value is nice, as well.
- Logic: Deletes all entries from the Completed or Failed buckets that are older than 7 days.
**Additional context**
Need to take into account the amount of storage that database will cost. And hence optimise it further. Documentation, related to the changes, related to env vars, and cron job along with the Database need to be added, to the Backend README. The current functionality should not be affected/broken.
- The issue is open to all for their versions of implementations.
Links:
https://github.com/etcd-io/bbolt
https://github.com/robfig/cron
コントリビューションガイド
調査の方向性
まず Go JobStore と NewJobQueue のエントリポイントを見つけ、次にバックエンドが /app/data の Docker ボリュームと .env の環境変数をどのように使用しているかを確認します。保留中、処理中、完了、失敗したジョブの永続化、スケジュールされたクリーンアップ、起動後の復旧を定義し、データベース、cron、環境変数の動作について Backend README を更新します。
索引モデルが issue の本文から書いたものです。
評価
- 技術スタック
- docker, go
- 領域
- backend, databases, documentation
- issue の種類
- 機能追加
- 難易度
- 5/5
- 見積もり時間
- 1週間以上
- 活発さ
- 停滞
- 明瞭さ
- おおむね明確
- 初心者へのやさしさ
- 35/100