Agent-Hellboy / Agent-Hellboy/ltm

Explore storage write-path and retrieval performance

Aperta
#6 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Go
Stelle
25
Fork
1
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

## Context

Follow-up to #5. After moving the collector transport to ringbuf and improving daemon-side batching, the next likely wins are in storage write-path profiling and read/query retrieval performance rather than more BPF transport work.

Temporary VM benchmarks on `INCVPS3-103.181.176.28` used a synthetic 300k-row SQLite event DB and measured common storage paths. No benchmark harness was committed.

## Findings

Write batching, 120k rows:

```text
batch=128 total_ms=5116.379 rows_per_sec=23454
batch=512 total_ms=4547.610 rows_per_sec=26387
batch=4096 total_ms=4082.857 rows_per_sec=29391
batch=16384 total_ms=3729.989 rows_per_sec=32172
```

Read/query paths over 300k rows:

```text
status p50_ms=107.682 p95_ms=112.039
tail_500 p50_ms= 3.737 p95_ms= 3.925
timeline_recent_200 p50_ms= 1.722 p95_ms= 1.943
timeline_recent_50000 p50_ms=477.720 p95_ms=510.994
category_action_write_200 p50_ms= 2.048 p95_ms= 2.706
pid_200 p50_ms= 1.057 p95_ms= 1.288
exact_path_200 p50_ms=226.658 p95_ms=240.600
path_like_prefix_200 p50_ms= 6.470 p95_ms= 6.719
query_text_marker_200 p50_ms=1284.768 p95_ms=1335.906
sockets_200 p50_ms=154.261 p95_ms=167.195
raw_count_by_category p50_ms= 51.452 p95_ms= 52.323
```

## Suspected Causes

- `Status()` scans whole-table aggregate, especially `SUM(dropped_before)`.
- `ExactPath` uses `(path = ? OR old_path = ?)`, but the schema only indexes `path`, not `old_path`.
- `Sockets()` filters `category = 'network'` and orders by timestamp, but the current index is `(category, action, ts)`, which is not ideal for this query.
- `QueryText` uses multi-column `LIKE '%term%'`, which is expected to full-scan; proper improvement likely needs FTS or a deliberate search index.
- Large timeline retrieval is dominated by materializing/scanning many full `Event` rows, including metadata/raw.

## Proposed Scope

First PR, low-risk storage indexes/query improvements:

- Add or evaluate an `old_path` index.
- Add or evaluate a `(category, ts)` index for socket/category timeline paths.
- Rework `Status()` to avoid full-table aggregate where possible, especially dropped totals.
- Benchmark before/after on a synthetic DB and a real recorded DB.
- Keep benchmark harness temporary or document manual commands, but do not commit noisy benchmark scaffolding unless it becomes a proper `go test -bench` package.

Separate later PR:

- Explore SQLite FTS for `QueryText`, or explicitly accept full-scan behavior and document it.

## Acceptance Criteria

- Existing unit tests pass: `go test ./...`.
- Integration still passes on Linux/root.
- PR body includes before/after query timings for at least `Status`, `ExactPath`, `Sockets`, and `QueryText` if touched.
- Any schema/index additions are reflected in generated schema docs and relevant contributor docs.

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.