kubescape / kubescape/operator

ContainerProfileWatch's startup relist re-queues scans for orphaned ReplicaSet generations, corrupting VulnerabilityManifestSummary on every operator restart

Open
#417 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
24
Forks
37
Avg merge
1d 14h
Merged PRs (30d)
9

Description

## Summary

`ContainerProfileWatch` re-lists and re-queues a scan for *every* `ContainerProfile` object in cluster storage on every operator process start (`watcher/containerprofilewatcher.go:50-62`) -- including profiles belonging to ReplicaSet generations that no longer exist. The default config leaves nothing to filter these out (`SkipProfilesWithoutInstances` defaults to `false` and isn't exposed in the Helm chart's `values.yaml`), and even the fallback pod lookup doesn't skip on failure (`getPod()` at line 139 just logs and continues if the pod is gone). Because the resulting `VulnerabilityManifestSummary` is keyed only by `--` (workload identity, not ReplicaSet), and the summary write path has no freshness/generation check, a scan of a months-old dead ReplicaSet's profile can silently overwrite the summary for the image that's actually running today.

## Evidence (live cluster, kubescape-operator chart 1.40.4, kubevuln v0.3.441)

`ContainerProfile` objects for one StatefulSet's container, none garbage-collected across 6 ReplicaSet generations spanning 3 months:

```
statefulset-alertmanager-kube-prometheus-stack-alertmanager-85fd7f467f-alertmanager-... 2026-06-15
statefulset-alertmanager-kube-prometheus-stack-alertmanager-5885f87586-alertmanager-... 2026-06-24 <- corrupted the summary
statefulset-alertmanager-kube-prometheus-stack-alertmanager-855c768ffd-alertmanager-... 2026-06-29
statefulset-alertmanager-kube-prometheus-stack-alertmanager-57548f9ff4-alertmanager-... 2026-07-15
statefulset-alertmanager-kube-prometheus-stack-alertmanager-77cfc9576-alertmanager-... 2026-07-15
statefulset-alertmanager-kube-prometheus-stack-alertmanager-5b6df87455-alertmanager-... 2026-08-31
statefulset-alertmanager-kube-prometheus-stack-alertmanager-9c6996f9f-alertmanager-... 2026-09-15 <- actually running
```

Operator log, right after a pod restart (triggered here by node churn during an EKS AMI rotation, but any operator restart reproduces it):

```
"scanning container profile","wlid":"wlid://cluster-.../namespace-monitoring/statefulset-alertmanager-kube-prometheus-stack-alertmanager",
"name":"statefulset-alertmanager-kube-prometheus-stack-alertmanager-5885f87586-config-reloader-7c63-5900"
```

`5885f87586` is the ReplicaSet hash replaced by five subsequent rollouts. After this scan ran, `VulnerabilityManifestSummary/statefulset-alertmanager-kube-prometheus-stack-alertmanager-alertmanager` reported `alertmanager:v0.32.2` -- several releases behind what was actually deployed (`v0.34.0`) -- and `vulnerabilitiesRef.relevant` pointed at the `5885f87586`-generation `VulnerabilityManifest`, not the current one. Deleting the summary and forcing a fresh scan while the operator was quiescent produced the correct result (`v0.34.0`, matching severities); the very next operator restart reintroduced the same corruption via the same mechanism.

## Why this is reachable, not hypothetical

Any event that restarts the operator pod triggers it: a node drain/replacement (confirmed reproduction), a chart upgrade, an OOM, a routine eviction. `ContainerProfileWatch`'s startup list (`containerprofilewatcher.go:50-62`) has no bound on age or on ReplicaSet liveness, and nothing elsewhere garbage-collects `ContainerProfile` objects when their owning ReplicaSet is deleted -- we found 6 generations going back 3 months for a single container.

## Suggested fixes (either resolves it; both together is more robust)

1. **Garbage-collect `ContainerProfile` objects when their owning ReplicaSet no longer exists** (owner references + GC, or a TTL/reconciler pass), so the startup relist has nothing stale to re-queue.
2. **Default `SkipProfilesWithoutInstances` to `true`** (or expose and default it in the Helm chart), and make `getPod()`'s failure path (`containerprofilewatcher.go:139-144`) skip dispatching the scan command rather than logging and continuing.
3. **Add a freshness/generation guard to the `VulnerabilityManifestSummary` write path**, so a scan result for a non-current ReplicaSet generation can't overwrite a summary that already reflects the current one -- the same class of fix already applied to `GetSBOM`'s one-directional version check in kubevuln#768 (a comparison that only guards one direction, so an out-of-order write silently wins).

## Environment

- kubescape-operator chart 1.40.4
- kubevuln v0.3.441 (confirmed current release at time of testing, not an outdated-pin issue)
- Reproduced across an EKS 1.34 and an EKS 1.35 cluster

Contributor guide

Open the contributing guide

Research direction

Start with watcher/containerprofilewatcher.go:50-62 and inspect getPod() at lines 139-144 to trace startup relisting and dispatch for profiles without live pods. Then locate the VulnerabilityManifestSummary write path and compare it with the freshness check referenced from kubevuln#768. Done means stale ReplicaSet generations cannot corrupt the summary after an operator restart, with coverage for the reported restart scenario.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, devops, security
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.