Agent-Hellboy / Agent-Hellboy/ltm

Explore storage write-path and retrieval performance

Offen
#6 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Go
Sterne
25
Forks
1
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

## Context

Follow-up to #5. After moving the collector transport to ringbuf and improving daemon-side batching, the next likely wins are in storage write-path profiling and read/query retrieval performance rather than more BPF transport work.

Temporary VM benchmarks on `INCVPS3-103.181.176.28` used a synthetic 300k-row SQLite event DB and measured common storage paths. No benchmark harness was committed.

## Findings

Write batching, 120k rows:

```text
batch=128 total_ms=5116.379 rows_per_sec=23454
batch=512 total_ms=4547.610 rows_per_sec=26387
batch=4096 total_ms=4082.857 rows_per_sec=29391
batch=16384 total_ms=3729.989 rows_per_sec=32172
```

Read/query paths over 300k rows:

```text
status p50_ms=107.682 p95_ms=112.039
tail_500 p50_ms= 3.737 p95_ms= 3.925
timeline_recent_200 p50_ms= 1.722 p95_ms= 1.943
timeline_recent_50000 p50_ms=477.720 p95_ms=510.994
category_action_write_200 p50_ms= 2.048 p95_ms= 2.706
pid_200 p50_ms= 1.057 p95_ms= 1.288
exact_path_200 p50_ms=226.658 p95_ms=240.600
path_like_prefix_200 p50_ms= 6.470 p95_ms= 6.719
query_text_marker_200 p50_ms=1284.768 p95_ms=1335.906
sockets_200 p50_ms=154.261 p95_ms=167.195
raw_count_by_category p50_ms= 51.452 p95_ms= 52.323
```

## Suspected Causes

- `Status()` scans whole-table aggregate, especially `SUM(dropped_before)`.
- `ExactPath` uses `(path = ? OR old_path = ?)`, but the schema only indexes `path`, not `old_path`.
- `Sockets()` filters `category = 'network'` and orders by timestamp, but the current index is `(category, action, ts)`, which is not ideal for this query.
- `QueryText` uses multi-column `LIKE '%term%'`, which is expected to full-scan; proper improvement likely needs FTS or a deliberate search index.
- Large timeline retrieval is dominated by materializing/scanning many full `Event` rows, including metadata/raw.

## Proposed Scope

First PR, low-risk storage indexes/query improvements:

- Add or evaluate an `old_path` index.
- Add or evaluate a `(category, ts)` index for socket/category timeline paths.
- Rework `Status()` to avoid full-table aggregate where possible, especially dropped totals.
- Benchmark before/after on a synthetic DB and a real recorded DB.
- Keep benchmark harness temporary or document manual commands, but do not commit noisy benchmark scaffolding unless it becomes a proper `go test -bench` package.

Separate later PR:

- Explore SQLite FTS for `QueryText`, or explicitly accept full-scan behavior and document it.

## Acceptance Criteria

- Existing unit tests pass: `go test ./...`.
- Integration still passes on Linux/root.
- PR body includes before/after query timings for at least `Status`, `ExactPath`, `Sockets`, and `QueryText` if touched.
- Any schema/index additions are reflected in generated schema docs and relevant contributor docs.

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.