kubescape / kubescape/operator
ContainerProfileWatch's startup relist re-queues scans for orphaned ReplicaSet generations, corrupting VulnerabilityManifestSummary on every operator restart
- Dominant language
- Go
- Stars
- 24
- Forks
- 37
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 9
Description
## Summary
`ContainerProfileWatch` re-lists and re-queues a scan for *every* `ContainerProfile` object in cluster storage on every operator process start (`watcher/containerprofilewatcher.go:50-62`) -- including profiles belonging to ReplicaSet generations that no longer exist. The default config leaves nothing to filter these out (`SkipProfilesWithoutInstances` defaults to `false` and isn't exposed in the Helm chart's `values.yaml`), and even the fallback pod lookup doesn't skip on failure (`getPod()` at line 139 just logs and continues if the pod is gone). Because the resulting `VulnerabilityManifestSummary` is keyed only by `--` (workload identity, not ReplicaSet), and the summary write path has no freshness/generation check, a scan of a months-old dead ReplicaSet's profile can silently overwrite the summary for the image that's actually running today.
## Evidence (live cluster, kubescape-operator chart 1.40.4, kubevuln v0.3.441)
`ContainerProfile` objects for one StatefulSet's container, none garbage-collected across 6 ReplicaSet generations spanning 3 months:
```
statefulset-alertmanager-kube-prometheus-stack-alertmanager-85fd7f467f-alertmanager-... 2026-06-15
statefulset-alertmanager-kube-prometheus-stack-alertmanager-5885f87586-alertmanager-... 2026-06-24 <- corrupted the summary
statefulset-alertmanager-kube-prometheus-stack-alertmanager-855c768ffd-alertmanager-... 2026-06-29
statefulset-alertmanager-kube-prometheus-stack-alertmanager-57548f9ff4-alertmanager-... 2026-07-15
statefulset-alertmanager-kube-prometheus-stack-alertmanager-77cfc9576-alertmanager-... 2026-07-15
statefulset-alertmanager-kube-prometheus-stack-alertmanager-5b6df87455-alertmanager-... 2026-08-31
statefulset-alertmanager-kube-prometheus-stack-alertmanager-9c6996f9f-alertmanager-... 2026-09-15 <- actually running
```
Operator log, right after a pod restart (triggered here by node churn during an EKS AMI rotation, but any operator restart reproduces it):
```
"scanning container profile","wlid":"wlid://cluster-.../namespace-monitoring/statefulset-alertmanager-kube-prometheus-stack-alertmanager",
"name":"statefulset-alertmanager-kube-prometheus-stack-alertmanager-5885f87586-config-reloader-7c63-5900"
```
`5885f87586` is the ReplicaSet hash replaced by five subsequent rollouts. After this scan ran, `VulnerabilityManifestSummary/statefulset-alertmanager-kube-prometheus-stack-alertmanager-alertmanager` reported `alertmanager:v0.32.2` -- several releases behind what was actually deployed (`v0.34.0`) -- and `vulnerabilitiesRef.relevant` pointed at the `5885f87586`-generation `VulnerabilityManifest`, not the current one. Deleting the summary and forcing a fresh scan while the operator was quiescent produced the correct result (`v0.34.0`, matching severities); the very next operator restart reintroduced the same corruption via the same mechanism.
## Why this is reachable, not hypothetical
Any event that restarts the operator pod triggers it: a node drain/replacement (confirmed reproduction), a chart upgrade, an OOM, a routine eviction. `ContainerProfileWatch`'s startup list (`containerprofilewatcher.go:50-62`) has no bound on age or on ReplicaSet liveness, and nothing elsewhere garbage-collects `ContainerProfile` objects when their owning ReplicaSet is deleted -- we found 6 generations going back 3 months for a single container.
## Suggested fixes (either resolves it; both together is more robust)
1. **Garbage-collect `ContainerProfile` objects when their owning ReplicaSet no longer exists** (owner references + GC, or a TTL/reconciler pass), so the startup relist has nothing stale to re-queue.
2. **Default `SkipProfilesWithoutInstances` to `true`** (or expose and default it in the Helm chart), and make `getPod()`'s failure path (`containerprofilewatcher.go:139-144`) skip dispatching the scan command rather than logging and continuing.
3. **Add a freshness/generation guard to the `VulnerabilityManifestSummary` write path**, so a scan result for a non-current ReplicaSet generation can't overwrite a summary that already reflects the current one -- the same class of fix already applied to `GetSBOM`'s one-directional version check in kubevuln#768 (a comparison that only guards one direction, so an out-of-order write silently wins).
## Environment
- kubescape-operator chart 1.40.4
- kubevuln v0.3.441 (confirmed current release at time of testing, not an outdated-pin issue)
- Reproduced across an EKS 1.34 and an EKS 1.35 cluster
Contributor guide
Research direction
Start with watcher/containerprofilewatcher.go:50-62 and inspect getPod() at lines 139-144 to trace startup relisting and dispatch for profiles without live pods. Then locate the VulnerabilityManifestSummary write path and compare it with the freshness check referenced from kubevuln#768. Done means stale ReplicaSet generations cannot corrupt the summary after an operator restart, with coverage for the reported restart scenario.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, devops, security
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100