Agent-Hellboy / Agent-Hellboy/ltm
Capture machine activity that eBPF syscall tracepoints cannot observe
- Langage dominant
- Go
- Étoiles
- 25
- Forks
- 1
- Métriques de merge des PR
- Aucune PR mergée en 30 j
Description
## Summary
`ltm` now hooks broad syscall, sched, and block tracepoints in `--mode ebpf` (see `ebpf/tracepoints_linux.go` and README eBPF coverage section). Several classes of machine activity still cannot be observed reliably through syscall tracepoints alone.
This issue tracks complementary collectors and hook types needed for fuller machine history.
## Gaps and proposed approaches
### 1. Kernel-internal work (no userspace syscall)
**Miss today:** writeback, journaling, kworker activity, driver-level work.
**Proposed hooks:**
- `tracepoint/writeback:*` for page-cache flush
- `tracepoint/ext4:*` / filesystem tracepoints where available
- CO-RE kprobes on VFS helpers (`vfs_read`, `vfs_write`, `do_sys_openat2`)
- kprobes on socket stack (`tcp_sendmsg`, `udp_sendmsg`)
### 2. Cached writes vs real disk I/O
**Miss today:** `write()` syscall shows app intent; actual disk hit may happen later.
**Proposed:**
- Correlate syscall `write` events with `block_rq_issue` (partially wired) and writeback tracepoints
- Query support: "when did this file actually hit disk?"
### 3. IPv6 sockets
**Miss today:** sockaddr parsing is IPv4-only (`sockaddr_in`).
**Proposed:**
- Parse `sockaddr_in6` in BPF for connect/bind/sendto/recvfrom
- Store full IPv6 addresses in `storage.Event`
### 4. Packet payloads
**Miss today:** metadata only (by design).
**Proposed (optional, off by default):**
- Small syscall buffer samples (first N bytes) for debug mode
- TC ingress/egress or XDP programs for L3/L4 packet metadata (not full payload capture by default)
### 5. io_uring fast path
**Miss today:** apps using io_uring may bypass classic read/write syscalls.
**Proposed hooks:**
- `io_uring:io_uring_submit_sqe`
- `io_uring:io_uring_complete`
- `sys_enter_io_uring_setup` / `sys_enter_io_uring_enter`
### 6. Remaining syscalls (~300+)
**Miss today:** curated high-value syscall list, not exhaustive.
**Proposed:**
- Optional `raw_syscalls/sys_enter` debug collector with BPF allowlist/denylist and sampling
### 7. XDP / TC / kprobes
**Miss today:** no packet-path or arbitrary kernel-function hooks.
**Proposed:**
- TC classifier for per-interface packet metadata
- XDP pass-through observer where NIC supports it
- CO-RE kprobe collector for VFS and network stack (kernel-version aware)
## Hard limits (likely out of scope)
- Decrypted TLS/HTTPS content without MITM
- GPU / RDMA / direct hardware paths
- Guest VM activity from host without a guest-side agent
- "Capture everything" at full rate on production without sampling
## Suggested implementation phases
1. IPv6 sockaddr parsing
2. io_uring tracepoint collector
3. writeback + block correlation for disk timing
4. optional `raw_syscalls` debug mode
5. TC / XDP network metadata collector
6. CO-RE kprobe VFS collector
## Related
- Commit expanding current eBPF coverage: 8b91d8a
- Hook table: `ebpf/tracepoints_linux.go`
---
## 8. Resource, pressure, and fault telemetry (sampling + discrete events)
**Miss today:** syscall/sched/block tracepoints show *activity* but not *machine
state*. During an incident (e.g. a one-minute hang) the timeline shows what ran,
but not why the box stalled. `mmap` events are not memory usage; `block_rq_issue`
alone cannot measure disk latency (it needs the matching completion). sysstat on
this VM samples only every 10 minutes — far too coarse to explain a one-minute
event.
**Missing evidence:**
- Per-process CPU, RSS, swap, I/O, run state, cgroup
- System CPU, load, run-queue depth, iowait
- Memory availability, reclaim, page faults, OOM kills
- CPU/memory/I/O pressure (PSI)
- Disk latency, completion, errors, queue depth
- Cgroup/container memory limits, throttling, OOM, restarts
- Hung tasks, soft lockups, kernel warnings
- NFS latency/failures; network drops, retransmits, interface failures
- systemd service failures/restarts
**Useful kernel sources (aggregate, do not store per-event):**
- `sched:sched_switch`, `sched_wakeup`, `sched_stat_blocked`, `sched_process_hang`
- `oom:mark_victim`
- `vmscan:mm_vmscan_direct_reclaim_*`, `mm_vmscan_throttled`
- `block:block_rq_complete`, `block_rq_error`, `block_rq_requeue` (pair with the
existing `block_rq_issue` for latency)
- `writeback:*`; IRQ / workqueue / thermal / power / cgroup events
- `/proc` + PSI (`/proc/pressure/*`) for cheap system/process sampling
**Proposed design — a separate low-frequency timeline, not "trace everything":**
- `system_samples` every ~1s: CPU, memory, swap, PSI, disk, network, run queue
- `process_samples` every ~5s: per-process CPU, RSS, swap, I/O, state, cgroup
- Discrete fault events, emitted immediately: OOM, hung task, disk error,
extreme I/O latency, container restart, kernel warning
- Keep the existing activity timeline to correlate "what commands/file/network
operations happened around the incident"
Capturing every scheduler event would overload the recorder, so these are
aggregated into samples plus rare discrete fault events.
**Prerequisite (done):** the self-capture feedback loop had to be fixed first —
`ltm status`/`query` reading the store was captured and dropped events, making
incident evidence incomplete. The recorder now filters its own comm in-kernel
and auto-ignores its DB/pid files (see `docs/security.md` → "Self-capture
guard"; CHANGELOG `[Unreleased]`).
### Suggested phases (append to the list above)
7. `/proc`+PSI system/process sampling timeline (`system_samples` / `process_samples`)
8. discrete fault events: OOM, hung-task, disk error, container restart, kernel warning
9. block completion correlation for disk latency (`block_rq_issue` → `block_rq_complete`)
Guide de contribution
Aucun guide de contribution indexé pour ce dépôt
Évaluation
Cette issue n'a pas encore été évaluée.