hoangsonww / hoangsonww/Claude-Code-Agent-Monitor
[Feature]: Trim hook payloads before storing events.data (whole-file mirrors grew my DB to 2 GB)
- Dominant language
- TypeScript
- Stars
- 1k
- Forks
- 234
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 18
Description
## What problem does this solve?
Claude Code PostToolUse payloads can duplicate whole files inside `events.data` (`tool_response.originalFile` on Edit/Write, `tool_response.file.content` or `.base64` on Read) and include large command output. On my Linux installation (346k events, 9.4k sessions), `events.data` occupied 1.78 GB of a 2.0 GB database; 1.47 GB was in PostToolUse rows, and individual rows exceeded 700 KB. At that point I observed routine dashboard requests saturating one CPU core and eventually a V8 out-of-memory failure.
The four largest PostToolUse contributors were Edit (1,026 MB), Read (261 MB), Bash (137 MB), and Write (22 MB). In the largest Edit row, `originalFile` accounted for 713 KB of 740 KB; the largest Read row contained a 672 KB base64 image. Only 137 MB was older than 30 days, while 628 MB came from the previous seven days, so age-based retention would miss the main cause.
I have a tested implementation on [my `improvements` branch](https://github.com/msshives-gif/Claude-Code-Agent-Monitor/tree/improvements). Would you be open to a PR using this storage-boundary design and these defaults?
## Proposed solution
Trim the payload once, right before `stmts.insertEvent.run(...)` in `server/routes/hooks.js`, with a small pure function. The in-memory `data` stays untouched for everything else the route does.
Live trimming applies at the Claude hook insert and to imported subagent tool events; live Codex ingestion is unchanged. The maintenance command described below scans every existing JSON `events.data` row, however, so it can also trim rows from other producers when they contain `tool_input` or `tool_response`.
1. Drop the whole-file mirrors, only for the native tools that produce them: `tool_response.originalFile` on Edit/MultiEdit/NotebookEdit/Write, `tool_response.file.content` and `.base64` on Read. An MCP tool that happens to return a short `file.content` keeps it.
2. Recursively cap string values inside `tool_input` and `tool_response` at `DASHBOARD_EVENT_STRING_CAP` UTF-16 units (default 2048), with a suffix recording how much was removed.
3. If a `tool_input` / `tool_response` is still larger than `DASHBOARD_EVENT_FIELD_CAP` bytes (default 16384; a whole-file `structuredPatch` on Write, say), keep only top-level strings of at most 200 characters, numbers, booleans, and nulls that fit the budget.
4. Record dropped, shortened, and replaced values under `data._trimmed`; the event detail pane labels this metadata.
Setting `DASHBOARD_EVENT_STRING_CAP=0` disables all trimming for newly stored rows, including mirror removal and the field cap; it cannot restore rows already trimmed.
Effect on my table, computed by running the trimmer over the existing rows: the 53,231 rows that change go from **1,461 MB to 145 MB**. New rows since enabling it top out around 3 KB.
A companion `npm run trim-events` applies the same rules to existing rows: read-only dry run by default, `--yes` rewrites in small id-ordered batches and VACUUMs, `--backup` snapshots the database with `VACUUM INTO` first. It reads the same `.env` and picks the same SQLite driver as the server, expects the dashboard to be stopped, and a re-run after an interruption is a no-op for rows already done.
Implementation: six commits on the linked branch, ending at `88a43cf`. It adds 22 test cases: 19 for the trimmer/hook path and 3 for the maintenance command.
### User-visible tradeoffs
- Stored event details become lossy previews. Native Edit/MultiEdit/NotebookEdit/Write `originalFile` fields and Read `file.content` / `file.base64` disappear; large diffs, command output, MCP responses, and other nested data may be truncated or replaced. Ordinary short paths and commands should remain, but expanded line counts — and, for unusual over-budget inputs, timeline titles — may reflect the preview. Claude Code's transcript files are not modified.
- The payload portion of `GET /api/events?q=` searches only the retained preview; `summary` and `tool_name` remain searchable.
- A restored export from before the change still holds the large rows; `npm run trim-events` cleans them.
## Alternatives considered
- **Age-based retention** — deletes history for little gain (see the age numbers above); the bloat is per-row size, not row count.
- **Trim in `scripts/hook-handler.js` instead of the server** — keeps bytes off the wire, but the installer embeds an absolute path to the handler in `~/.claude/settings.json`; trimming there would also miss other producers such as history import and forwarded remote hooks. Server-side covers every producer. Both could coexist later.
- **Drop `tool_response` entirely** — loses the diff and terminal previews the detail pane does use.
## Area
Server / API
Contributor guide
Assessment
This issue has not been assessed yet.