CCExtractor / CCExtractor/ccsync

Persistent Job Queue & Maintenance Worker for backend

オープン
#367 コメント 2 件 リアクション 0 件 担当者 0 名 GitHub で見る
backend complex enhancement
主要言語
TypeScript
スター
37
フォーク
76
平均マージ
3日 7時間
マージ済み PR(30日)
2

説明

**Is your feature request related to a problem? Please describe.**
The current job queue is volatile. If the backend container crashes or restarts, all queued tasks are lost forever. This needs to be persistent, so that occasional backend downtimes do not affect the Jobs. Additionally, as we implement persistence, we need a way to manage the lifecycle of the "Success" and "Failure" logs stored in the database to prevent unbounded disk usage.

**Describe the solution you'd like**
1. Implement a disk-backed queue using `bbolt` (or something similar, open to suggestions) and an internal Go-based cron scheduler for maintenance.
- The Queue will work as a database that will store all Jobs from the JobStore.

2. Persistence Layer (bbolt)
- Store the database file at /app/data/queue.db to use the existing Docker volume.
- Job States: Implement buckets or tables for Pending, In-Progress (doubt if it should exist, perhaps Pending, Completed, and Failed should suffice for our use case), and Completed statuses.
- Reliability: On startup, a NewJobQueue function must scan the Pending and In-Progress buckets and re-insert those jobs into the channel. Upon completion, mark them as completed.

3. Scheduled Maintenance (Cron)
- Integrate a package like robfig/cron/v3 to handle periodic tasks without external dependencies.
- Cleanup Job: Runs every 24 hours (or at midnight). Add that CRON parameter as an optional variable in .env file. Users might want to customise, but having a default value is nice, as well.
- Logic: Deletes all entries from the Completed or Failed buckets that are older than 7 days.

**Additional context**
Need to take into account the amount of storage that database will cost. And hence optimise it further. Documentation, related to the changes, related to env vars, and cron job along with the Database need to be added, to the Backend README. The current functionality should not be affected/broken.

- The issue is open to all for their versions of implementations.

Links:
https://github.com/etcd-io/bbolt
https://github.com/robfig/cron

コントリビューションガイド

コントリビューションガイドを開く

調査の方向性

まず Go JobStore と NewJobQueue のエントリポイントを見つけ、次にバックエンドが /app/data の Docker ボリュームと .env の環境変数をどのように使用しているかを確認します。保留中、処理中、完了、失敗したジョブの永続化、スケジュールされたクリーンアップ、起動後の復旧を定義し、データベース、cron、環境変数の動作について Backend README を更新します。

索引モデルが issue の本文から書いたものです。

評価

技術スタック
docker, go
領域
backend, databases, documentation
issue の種類
機能追加
難易度
5/5
見積もり時間
1週間以上
活発さ
停滞
明瞭さ
おおむね明確
初心者へのやさしさ
35/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。