Agent-Hellboy / Agent-Hellboy/ltm
Explore storage write-path and retrieval performance
- 主要语言
- Go
- 星标
- 25
- 派生
- 1
- PR 合并指标
- 30 天内没有已合并 PR
描述
## Context
Follow-up to #5. After moving the collector transport to ringbuf and improving daemon-side batching, the next likely wins are in storage write-path profiling and read/query retrieval performance rather than more BPF transport work.
Temporary VM benchmarks on `INCVPS3-103.181.176.28` used a synthetic 300k-row SQLite event DB and measured common storage paths. No benchmark harness was committed.
## Findings
Write batching, 120k rows:
```text
batch=128 total_ms=5116.379 rows_per_sec=23454
batch=512 total_ms=4547.610 rows_per_sec=26387
batch=4096 total_ms=4082.857 rows_per_sec=29391
batch=16384 total_ms=3729.989 rows_per_sec=32172
```
Read/query paths over 300k rows:
```text
status p50_ms=107.682 p95_ms=112.039
tail_500 p50_ms= 3.737 p95_ms= 3.925
timeline_recent_200 p50_ms= 1.722 p95_ms= 1.943
timeline_recent_50000 p50_ms=477.720 p95_ms=510.994
category_action_write_200 p50_ms= 2.048 p95_ms= 2.706
pid_200 p50_ms= 1.057 p95_ms= 1.288
exact_path_200 p50_ms=226.658 p95_ms=240.600
path_like_prefix_200 p50_ms= 6.470 p95_ms= 6.719
query_text_marker_200 p50_ms=1284.768 p95_ms=1335.906
sockets_200 p50_ms=154.261 p95_ms=167.195
raw_count_by_category p50_ms= 51.452 p95_ms= 52.323
```
## Suspected Causes
- `Status()` scans whole-table aggregate, especially `SUM(dropped_before)`.
- `ExactPath` uses `(path = ? OR old_path = ?)`, but the schema only indexes `path`, not `old_path`.
- `Sockets()` filters `category = 'network'` and orders by timestamp, but the current index is `(category, action, ts)`, which is not ideal for this query.
- `QueryText` uses multi-column `LIKE '%term%'`, which is expected to full-scan; proper improvement likely needs FTS or a deliberate search index.
- Large timeline retrieval is dominated by materializing/scanning many full `Event` rows, including metadata/raw.
## Proposed Scope
First PR, low-risk storage indexes/query improvements:
- Add or evaluate an `old_path` index.
- Add or evaluate a `(category, ts)` index for socket/category timeline paths.
- Rework `Status()` to avoid full-table aggregate where possible, especially dropped totals.
- Benchmark before/after on a synthetic DB and a real recorded DB.
- Keep benchmark harness temporary or document manual commands, but do not commit noisy benchmark scaffolding unless it becomes a proper `go test -bench` package.
Separate later PR:
- Explore SQLite FTS for `QueryText`, or explicitly accept full-scan behavior and document it.
## Acceptance Criteria
- Existing unit tests pass: `go test ./...`.
- Integration still passes on Linux/root.
- PR body includes before/after query timings for at least `Status`, `ExactPath`, `Sockets`, and `QueryText` if touched.
- Any schema/index additions are reflected in generated schema docs and relevant contributor docs.
贡献指南
这个仓库没有索引到贡献指南
评估
这个 Issue 还没有评估数据。