kubescape / kubescape/node-agent

Syscall observations are silently lost between the advise_seccomp map and the container profile (87% loss measured)

Open
#918 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
37
Forks
28
Avg merge
23h 31m
Merged PRs (30d)
29

Description

Follow-up to #910/#912 with hard measurements. Generated seccomp profiles are missing syscalls the kernel demonstrably recorded, which is the root of the "generated profile kills the workload" class (#912).

## Method

A probe container runs a single python process (no fork/exec) issuing distinct, rare syscalls at known times: chdir+mkdir at t=0, rename t+10s, link t+25s, symlink t+45s, rmdir t+70s, socket t+100s, then pure sleep. In parallel, a privileged pod dumps the `syscalls_per_mntns` map (bpftool, 1s interval) for the probe mntns, so kernel ground truth is captured before and between the 30s `BPF_MAP_LOOKUP_AND_DELETE_BATCH` drains. Learning period 5m. node-agent vendored at v0.3.179.

## Kernel ground truth (measured)

The eBPF side is flawless. The map entry for the probe mntns contained, drain window by drain window:

| window | contents |
|---|---|
| 0 (container start) | 62 syscalls: full startup incl. execve, brk, read, mmap, chdir, rename + footer byte |
| 1 | chdir, mkdir, rename (markers t0/t10) |
| 2 | link, symlink (t25/t45) |
| 3 | rmdir (t70) |
| 4 | socket (t100) |
| 5-6 | clock_nanosleep (steady state) |

Union: 67 distinct syscalls. Every marker present. The deliberate no-container-filtering design in `program.bpf.c` works exactly as intended.

## What the profile contained

9 syscalls: clock_nanosleep, close, fstat, getdents64, lseek, openat, prctl, tgkill, write.

58 of 67 lost (87%). All 6 markers lost. execve, read, mmap, brk lost. The losses are not a time window (they span every drain), and not whole-packet (survivors are a strict subset of window 0 plus repeats). node-agent logged zero errors or warnings during the run: no queue-full drops, no enqueue failures, no splits.

## Where it is NOT

Ruled out with evidence on the same cluster:
- eBPF/kernel recording (table above)
- container attach latency (attach within 1s of create; losses continue for minutes)
- OrderedEventQueue overflow (0 "Event queue full" in logs)
- profile queue drop/split/413 (0 occurrences; split fix from #866 present)
- storage-side merge (mergeContainerProfileTS appends + dedupes)

Remaining scope: the userspace path from the mapiter datasource emission through `decodeSyscalls`/`eventCallback`/`ReportSyscall` into containerData. Note `ReportSyscall` errors with `ErrContainerNotFound` are swallowed silently (`logEventError`), so any attribution failure there is invisible.

## Why this is severe

The drain deletes the map entry, so each observation exists exactly once; whatever this path drops is unrecoverable. A workload that starts once and idles (most servers) gets a profile missing its startup and one-shot syscalls, and a seccomp profile generated from it prevents the container from starting (#912) or breaks it at runtime.

## Suggested directions

1. Make the syscalls pipeline idempotent instead of once-only: stop draining (plain lookup instead of lookup-and-delete, entries removed on container removal), so every fetch re-emits the cumulative bitmap and any single loss self-heals on the next fetch. The bitmap is 501 bytes per mntns; re-emission cost is negligible and downstream is already a dedup/union.
2. Make every drop in the userspace path loud: `ReportSyscall` attribution failures must at minimum increment a metric, not vanish.
3. Keep a completeness guard in generators regardless (armosec side: runtime baseline union), because a delta pipeline can never prove completeness.

Full repro manifests and the bpftool watcher script available on request.

Contributor guide

Open the contributing guide

Research direction

Trace the userspace path from the mapiter datasource through decodeSyscalls, eventCallback, ReportSyscall, and containerData; inspect program.bpf.c and mergeContainerProfileTS for the producer and merge behavior. Reproduce the measured drain sequence with the available manifests and bpftool watcher, then instrument attribution failures currently swallowed by logEventError. Done means the observed syscall union reaches the generated profile reliably and drops are visible through a metric or error signal.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes, linux
Domain
security
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.