Implement WatchList / sendInitialEvents protocol support
- Dominant language
- Go
- Stars
- 4
- Forks
- 22
- Avg merge
- 20h 43m
- Merged PRs (30d)
- 33
Description
## Context
This is the follow-up implementation issue for #318, which reported that Rancher agents on k3s clusters enter a tight retry loop ("very short watch" error) when kubescape storage is installed alongside a cluster that has the `WatchList` feature gate enabled.
As an immediate mitigation, `WatchList=false` was added alongside the existing `ServerSideApply=false` in `pkg/cmd/server/start.go`. This prevents clients from using WatchList semantics against this server. Full protocol support is tracked here.
## Root Causes (from trace investigation)
Three converging gaps were found in the file-based storage backend:
### 1. Missing terminal BOOKMARK in `StorageImpl.Watch`
**File:** `pkg/registry/file/storage.go:388-401`
`StorageImpl.Watch` ignores `opts.SendInitialEvents` entirely. When a WatchList-aware client sends `watch?sendInitialEvents=true`, the server accepts the request but never emits the required terminal `BOOKMARK` event with annotation `k8s.io/initial-events-end: "true"`.
The standard Kubernetes `Cacher` (etcd wrapper in `k8s.io/apiserver/pkg/storage/cacher`) is the component that injects this bookmark — but it is not in the call chain here. The server wires `DryRunnableStorage{Storage: storageImpl}` directly, bypassing the `Cacher`. The HTTP watch handler (`endpoints/handlers/watch.go`) only relays BOOKMARK events from storage; it never injects them.
**Required change:** `StorageImpl.Watch` must check `opts.SendInitialEvents`, replay the current object list as `ADDED` events, then emit a `BOOKMARK{k8s.io/initial-events-end: "true"}` event before streaming live events.
### 2. `immutableStorage.Watch` returns a pre-closed channel
**File:** `pkg/registry/file/storage.go:1221-1222`
`immutableStorage.Watch` unconditionally returns `watch.NewEmptyWatch()` — a channel that is immediately closed. This affects the following resource types:
- `GeneratedNetworkPolicy` (`pkg/registry/file/generatednetworkpolicy.go:26`)
- Likely `ConfigurationScanSummary` and `VulnerabilitySummary` (also embed `immutableStorage`)
A pre-closed channel gives a WatchList reflector 0 items in < 1 second — the exact trigger for the "very short watch" tight retry loop. The old List+Watch reflector tolerated this (it interleaved a List between retries); the WatchList reflector does not.
**Required change:** Either implement a real watch for these resource types, or explicitly signal WatchList unsupported so the reflector falls back gracefully.
### 3. Namespace-scoped watch rejection in `StorageImpl.Watch`
**File:** `pkg/registry/file/storage.go:393-396`
`StorageImpl.Watch` returns `watch.NewEmptyWatch()` for any namespace-scoped watch key (there is a `// FIXME find an alternative to fix NS deletion` comment on this path). This produces the same pre-closed channel symptom as gap #2 for namespace-scoped resources.
**Required change:** Resolve the underlying namespace deletion concern and implement proper namespace-scoped watch support.
## Acceptance Criteria
- [ ] `StorageImpl.Watch` reads `opts.SendInitialEvents` and emits a terminal `BOOKMARK{k8s.io/initial-events-end: "true"}` after replaying the initial list
- [ ] `immutableStorage.Watch` returns a functional (non-immediately-closed) watch, or correctly signals WatchList unsupported
- [ ] Namespace-scoped watches on `StorageImpl` return a real watch (not `NewEmptyWatch()`)
- [ ] `WatchList=false` gate override in `start.go` is removed once all three gaps are fixed
- [ ] Integration test or documented manual test confirming Rancher/reflector compatibility
Contributor guide
Research direction
Start in pkg/registry/file/storage.go, especially StorageImpl.Watch around lines 388-401 and immutableStorage.Watch around lines 1221-1222, then inspect pkg/cmd/server/start.go and the watch handler. Trace how SendInitialEvents, namespace-scoped watches, and immutable resources are handled. Done means all three watch paths work or explicitly reject WatchList, the WatchList=false override is removed, and an integration or documented manual test confirms reflector compatibility.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- api, backend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100