hoangsonww / hoangsonww/Claude-Code-Agent-Monitor
[Feature]: Tiered retention with rollup-before-delete so history shrinks instead of disappearing
- Dominant language
- TypeScript
- Stars
- 1k
- Forks
- 234
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 18
Description
### What problem does this solve?
The database grows without bound and the only tools for managing that are blunt. `POST /api/settings/cleanup` and `ccam cleanup` delete old rows; `ccam clear-data` deletes everything. Both are all-or-nothing on the *raw* data, which forces a bad trade: keep every payload forever and watch `dashboard.db` grow, or delete the old months and lose the long-run analytics that make the Analytics page worth having.
For anyone running agents daily this bites within months. `events.data` holds full tool inputs and responses and is by far the largest column in the schema — but the value of a six-month-old `PostToolUse` payload is near zero, while the value of "how many events, of which types, at what cost, on which project, that week" is high and permanent. There is currently no way to keep the second and drop the first.
The knock-on effects are real, not theoretical: full-table scans get slower (see #224, the desktop freeze on a large `~/.claude` history), exports get unwieldy (#251), and users end up periodically nuking their history — which silently ruins every long-range trend the dashboard draws.
### Proposed solution
**Tiered retention with rollup-before-delete**, configurable in Settings, so old data is compacted rather than lost.
**Tier model** — three age bands, each with a user-set boundary and sensible defaults:
| Tier | Default age | What is kept |
| --- | --- | --- |
| Hot | 0–30 days | Everything, unchanged |
| Warm | 30–180 days | Rows kept, `events.data` payloads dropped (summary, type, tool, timestamps retained) |
| Cold | 180+ days | Raw rows removed, replaced by daily rollups |
**Rollups** — before anything is deleted, a summarizer writes daily aggregates into a new additive table:
```sql
CREATE TABLE IF NOT EXISTS activity_rollups (
day TEXT NOT NULL,
project TEXT,
provider TEXT,
model TEXT,
event_count INTEGER NOT NULL DEFAULT 0,
session_count INTEGER NOT NULL DEFAULT 0,
tool_counts TEXT, -- JSON: {"Bash":128,"Edit":94,...}
input_tokens INTEGER NOT NULL DEFAULT 0,
output_tokens INTEGER NOT NULL DEFAULT 0,
cache_read_tokens INTEGER NOT NULL DEFAULT 0,
cache_write_tokens INTEGER NOT NULL DEFAULT 0,
cost_usd REAL NOT NULL DEFAULT 0,
PRIMARY KEY (day, project, provider, model)
);
```
Analytics queries then read raw rows for recent ranges and rollups for older ones, unioning the two so a 12-month trend still renders after the raw events are gone.
**Optional cold archive** — before deletion, dump the affected rows to a compressed JSONL bundle under the data dir, in the same shape `POST /api/settings/import` already accepts. That makes the whole thing reversible: the data leaves SQLite but not the machine, and it can be re-imported into a scratch instance if someone genuinely needs a two-year-old payload.
**Safety** — this is destructive by design, so it needs the project's usual guardrails: disabled by default, an explicit opt-in per tier, a mandatory dry-run (`ccam retention --dry-run` / a preview in Settings) reporting exactly how many rows and how many MB each tier would affect, `VACUUM` only after a successful pass, and a hard rule that a rollup must be written and verified before the corresponding raw rows are deleted.
**Surfacing** — Settings shows current DB size broken down by table, the projected size after applying the policy, and when the last retention pass ran. `ccam retention status|run|--dry-run` for headless boxes.
### Alternatives considered
- **`ccam cleanup` on a cron.** What people do now. It deletes the analytics along with the payloads — the trend lines just end.
- **Payload compression instead of deletion.** Helps (`events.data` is very compressible) and is complementary, but it's a constant-factor win against unbounded growth. It also can't be combined with #148-style redaction as cleanly as tiering can.
- **Move old data to a separate archive database file.** Similar effect but more moving parts than a rollup table, and it complicates every analytics query rather than just the long-range ones.
- **Do nothing and let users manage it.** The default outcome is a slow dashboard and, eventually, someone running `clear-data` — which is the worst version of retention.
### Area
Database / SQLite
### How important is this to you?
Would significantly improve my workflow
### Additional context
Related to but distinct from #149 (backup/restore/sync) — that's about moving a complete dataset between machines, this is about bounding what a single machine keeps. They share the archive/import format, and building the cold-archive bundle on top of the existing export shape would let both features reuse one serializer.
Contributor guide
Assessment
This issue has not been assessed yet.