DB record retention management layer
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 15h 13m
- Merged PRs (30d)
- 368
Description
Backend.AI accumulates log / history / lifecycle records indefinitely, but automated retention sweeping today exists only for `error_logs` (log_cleanup_timer.py, a GlobalTimer firing every 20s, reading etcd key `config/logs/error/retention`, default 90d). `sessions` / `kernels` / Redis stats are purged only by the manual `clear-history` CLI (default 1yr, operator-run).
This Epic introduces a \***general-purpose retention management layer**\*: a super-admin defines a retention period per data category, and the manager periodically purges records older than that. Targets include not only pure append-only logs (event_logs, session/kernel scheduling_history, deployment_history, route_history, replica_group_history, login_history, audit_logs, usage buckets) but also lifecycle records such as terminated sessions/kernels that today are removed only by the clear-history CLI. The layer generalizes the log_cleanup_timer sweep pattern, standardizes policy storage / admin exposure / sweep execution, and absorbs the responsibilities of the clear-history CLI.
Core challenge: the data is heterogeneous (pure logs vs lifecycle session/kernel records vs billing-relevant usage buckets), so the central design question is how to manage them consistently. That management model is decided first via a BEP (see the Write BEP task).
Related existing epics to cross-link in the BEP (adjacent, not duplicate): BA-4383 (artifact storage retention), BA-3037 (unused docker image retention).
JIRA Issue: BA-6924
Contributor guide
Research direction
Start with log_cleanup_timer.py and the clear-history CLI to understand the existing retention paths and their data categories. Then review the Write BEP task and related epics BA-4383 and BA-3037. Done means an agreed BEP defining the general retention model, policy storage, admin exposure, and sweep responsibilities across the listed records.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, redis
- Domain
- api, backend, cli, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100