kubescape / kubescape/node-agent

Learned seccomp profile can be marked complete and auto-enforced while missing syscalls the workload needs, with no self-healing path

Open
#912 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
37
Forks
28
Avg merge
23h 31m
Merged PRs (30d)
29

Description

## Summary

A learned seccomp profile can be marked `completed`/`complete` and auto-enforced onto its workload while still missing syscalls the workload genuinely needs — and once enforced, that gap can never self-heal through the normal learn → enforce pipeline, because enforcement denial happens *before* the same eBPF gadget's own observation hook fires. This causes silent, permanent, hard-to-diagnose functional breakage in profiled workloads.

## Live reproduction

Deployed a stock `docker.io/hisu/ping-app` (Apache 2.4.54 + PHP 7.4.33) workload with a short custom `kubescape.io/max-sniffing-time: 15m` learning window. Kubescape learned and auto-enforced a `Localhost` seccomp profile (`SCMP_ACT_ERRNO` default, ~85 allowed syscalls) back onto the same deployment.

With that profile enforced, every real HTTP request to the app failed:

```
(1)Operation not permitted: [client ...] AH00036: access to /index.php failed (filesystem path '/var')
127.0.0.1 - - [...] "GET / HTTP/1.1" 200 0 "-" "curl/8.14.1"
```

Apache returns `200` but with an empty body — the failure is silent from the client's perspective. Reproduced identically on two independent clusters (different node-agent deployment modes: local CRD storage and backend-storage mode), ruling out a storage-mode-specific cause.

Root cause confirmed via kernel audit log (profile temporarily switched to `SCMP_ACT_LOG` for diagnosis — logs but doesn't block):

```
audit: type=1326 ... comm="apache2" ... syscall=6 ... (lstat)
audit: type=1326 ... comm="apache2" ... syscall=7 ... (poll)
audit: type=1326 ... comm="apache2" ... syscall=20 ... (writev)
```

`lstat`, `poll`, and `writev` are genuinely used by Apache/PHP during normal request handling but were never captured during the 15-minute learning window (most likely because whatever traffic hit the pod during learning — e.g. only TCP-level liveness probes — never exercised the real page-serving code path). The generated `ApplicationProfile`/`SeccompProfile` was nonetheless marked `kubescape.io/completion: complete`.

## Why this can never self-heal

The `advise_seccomp` gadget observes syscalls via a `raw_tracepoint/sys_enter` hook. Per standard kernel syscall-entry ordering, `__secure_computing()` (where seccomp's `SCMP_ACT_ERRNO` denial happens) runs *before* `trace_sys_enter()` fires. A syscall denied by the enforced profile therefore never reaches the same observation mechanism that's supposed to expand/correct the profile. Once a profile with any gap gets enforced, the missing syscalls become permanently invisible to the exact pipeline that could otherwise discover and fix them — verified live: the container's cumulative captured syscall count (`resource_size`) continued growing normally to completion under full enforcement (202 → 322 → 325 over ~17 minutes), confirming only the specifically-denied syscalls are affected, not general observation.

## Impact

- `completion: complete` / `status: completed` asserts a guarantee the learning-window heuristic can't actually back up — it means the time window closed, not that every code path was exercised.
- Once such a profile is enforced, the workload can suffer silent, intermittent, hard-to-diagnose failures (in our case: Apache returning empty `200` responses) with no way for the system itself to detect or repair the gap.
- Shortening the learning window (e.g. via `kubescape.io/max-sniffing-time`) makes this substantially more likely to bite, but the underlying risk exists for any learned-then-enforced profile whose learning traffic didn't cover every real code path.

## Suggested directions (not prescriptive)

- Don't treat "learning window closed" as equivalent to "behaviorally complete" — consider a distinct signal/gate before auto-enforcing a profile (e.g. traffic-coverage heuristics, canary/audit period before switching from `SCMP_ACT_LOG`-style non-blocking mode to `SCMP_ACT_ERRNO`).
- Consider detecting EPERM-driven failures on an enforced workload and automatically reopening/widening the profile rather than requiring manual intervention.
- At minimum, surface this limitation prominently wherever `completion: complete` is shown, so it isn't read as a functional-completeness guarantee.

## Environment

- `kubescape/node-agent` v0.3.179 (vendored via `armosec/private-node-agent`)
- IG `advise_seccomp` gadget (`github.com/matthyx/inspektor-gadget` fork)
- Reproduced on two DigitalOcean K8s clusters, containerd 2.2.3, kernel 6.12.96, x86_64
- Related: kubescape/node-agent#910 (separate decode bug that pollutes generated profiles with a bogus `"unknown"` syscall entry — not the cause of this issue, but found during the same investigation)

Contributor guide

Open the contributing guide

Research direction

Start with the advise_seccomp gadget's raw_tracepoint/sys_enter observation path and the completion-to-auto-enforcement flow. Review how the learning-window signal relates to the cited lstat, poll, and writev denials; done should include a decided safeguard against unsafe enforcement or a clear limitation signal, with the no-self-healing behavior addressed.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
security
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.