kubescape / kubescape/storage

Implement WatchList / sendInitialEvents protocol support

Open
#320 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
4
Forks
22
Avg merge
20h 43m
Merged PRs (30d)
33

Description

## Context

This is the follow-up implementation issue for #318, which reported that Rancher agents on k3s clusters enter a tight retry loop ("very short watch" error) when kubescape storage is installed alongside a cluster that has the `WatchList` feature gate enabled.

As an immediate mitigation, `WatchList=false` was added alongside the existing `ServerSideApply=false` in `pkg/cmd/server/start.go`. This prevents clients from using WatchList semantics against this server. Full protocol support is tracked here.

## Root Causes (from trace investigation)

Three converging gaps were found in the file-based storage backend:

### 1. Missing terminal BOOKMARK in `StorageImpl.Watch`
**File:** `pkg/registry/file/storage.go:388-401`

`StorageImpl.Watch` ignores `opts.SendInitialEvents` entirely. When a WatchList-aware client sends `watch?sendInitialEvents=true`, the server accepts the request but never emits the required terminal `BOOKMARK` event with annotation `k8s.io/initial-events-end: "true"`.

The standard Kubernetes `Cacher` (etcd wrapper in `k8s.io/apiserver/pkg/storage/cacher`) is the component that injects this bookmark — but it is not in the call chain here. The server wires `DryRunnableStorage{Storage: storageImpl}` directly, bypassing the `Cacher`. The HTTP watch handler (`endpoints/handlers/watch.go`) only relays BOOKMARK events from storage; it never injects them.

**Required change:** `StorageImpl.Watch` must check `opts.SendInitialEvents`, replay the current object list as `ADDED` events, then emit a `BOOKMARK{k8s.io/initial-events-end: "true"}` event before streaming live events.

### 2. `immutableStorage.Watch` returns a pre-closed channel
**File:** `pkg/registry/file/storage.go:1221-1222`

`immutableStorage.Watch` unconditionally returns `watch.NewEmptyWatch()` — a channel that is immediately closed. This affects the following resource types:
- `GeneratedNetworkPolicy` (`pkg/registry/file/generatednetworkpolicy.go:26`)
- Likely `ConfigurationScanSummary` and `VulnerabilitySummary` (also embed `immutableStorage`)

A pre-closed channel gives a WatchList reflector 0 items in < 1 second — the exact trigger for the "very short watch" tight retry loop. The old List+Watch reflector tolerated this (it interleaved a List between retries); the WatchList reflector does not.

**Required change:** Either implement a real watch for these resource types, or explicitly signal WatchList unsupported so the reflector falls back gracefully.

### 3. Namespace-scoped watch rejection in `StorageImpl.Watch`
**File:** `pkg/registry/file/storage.go:393-396`

`StorageImpl.Watch` returns `watch.NewEmptyWatch()` for any namespace-scoped watch key (there is a `// FIXME find an alternative to fix NS deletion` comment on this path). This produces the same pre-closed channel symptom as gap #2 for namespace-scoped resources.

**Required change:** Resolve the underlying namespace deletion concern and implement proper namespace-scoped watch support.

## Acceptance Criteria

- [ ] `StorageImpl.Watch` reads `opts.SendInitialEvents` and emits a terminal `BOOKMARK{k8s.io/initial-events-end: "true"}` after replaying the initial list
- [ ] `immutableStorage.Watch` returns a functional (non-immediately-closed) watch, or correctly signals WatchList unsupported
- [ ] Namespace-scoped watches on `StorageImpl` return a real watch (not `NewEmptyWatch()`)
- [ ] `WatchList=false` gate override in `start.go` is removed once all three gaps are fixed
- [ ] Integration test or documented manual test confirming Rancher/reflector compatibility

Contributor guide

Open the contributing guide

Research direction

Start in pkg/registry/file/storage.go, especially StorageImpl.Watch around lines 388-401 and immutableStorage.Watch around lines 1221-1222, then inspect pkg/cmd/server/start.go and the watch handler. Trace how SendInitialEvents, namespace-scoped watches, and immutable resources are handled. Done means all three watch paths work or explicitly reject WatchList, the WatchList=false override is removed, and an integration or documented manual test confirms reflector compatibility.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
api, backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.