apache / apache/apisix-dashboard

etcd watch leak during store reinitialization (read-only dashboard exhausts etcd memory/CPU)

Open
#3,411 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
1.2k
Forks
617
Avg merge
4d 18h
Merged PRs (30d)
1

Description

## Summary

APISIX Dashboard 3.0.x can leak etcd logical watchers when store watches fail and are reinitialized — an application-level watch lifecycle leak. A **read-only** dashboard can exhaust etcd memory/CPU with no high-rate writes and a tiny dataset.

## Environment

- `apache/apisix-dashboard` 3.0.0 (`manager-api` 3.0.0); checked `v3.0.1` has the same relevant code.
- Embedded `go.etcd.io/etcd/client/v3` v3.5.5, connecting directly to etcd.

## Root cause (`api/internal/core/store/store.go`, `api/cmd/root.go`)

1. A failed watch loop appends its `GenericStore` to a global `storeNeedReInit` slice — **no dedup, never cleared, no mutex**.
2. `ReInit()` runs ~every 2 minutes, iterates the slice, and calls `store.Init()`.
3. `GenericStore.Init()` → `listAndWatch()` overwrites `s.cancel` **without cancelling the previous watch context**.
4. The health checker also force-calls `Init()` per store on transient etcd issues.

With repeated `required revision has been compacted` / transient errors, live logical watchers accumulate in the Go client. Because `clientv3` multiplexes logical watchers over a few gRPC Watch streams, etcd shows a distinctive signature: low `watch_stream_total`, ~flat `grpc_server_started_total{Watch}`, but rapidly rising `grpc_server_msg_received_total{Watch}` and `etcd_debugging_mvcc_watcher_total`.

## Production evidence

- Dashboard RSS grew to ~46 GiB; no restart for 186 days.
- The connected etcd member reached **millions** of watchers, ~25 CPU cores in GC, and OOM/restart loops.
- Dashboard logs: repeated `etcd watch error: key: /apisix/... err: etcdserver: mvcc: required revision has been compacted` followed by `etcd watch exception closed, restarting: resource: ...`.
- **Scaling the dashboard to 0 immediately stopped the WatchCreate storm; etcd `watcher_total` dropped back to single digits.**

## Proposed fix

Make reinitialization idempotent and cancel superseded watches:

- Replace the global slice with a mutex-protected **set**; **drain** it in `ReInit()`; **requeue only failed** entries.
- Assign a **generation** to each watch; only enqueue reinit when the current generation exits unexpectedly (`generation == current && !closing && ctx.Err() == nil`). This avoids misclassifying deliberate cancels and avoids re-enqueue churn.
- `Close()` must remove the store from the pending set and mark it closed; `Init()`/`ReInit()` must check closed so a closed store is not resurrected.
- Cancel the superseded watch (either cancel-old-then-relist, or new-then-cancel-old). If you keep new-before-cancel, guard event apply by revision so late events from the old watch can't overwrite cache during the dual-watch window.
- Add **per-store backoff** for permanently-failing reinit to avoid a tight 2-minute retry + log spam.

## Impact

Production stability: a read-only dashboard can exhaust the shared etcd's memory/CPU. Upgrading 3.0.0 → 3.0.1 is **not** a fix (same code). We mitigated by scaling the dashboard to zero.

Contributor guide

Open the contributing guide

Research direction

Start with api/internal/core/store/store.go and api/cmd/root.go, tracing GenericStore.Init(), listAndWatch(), Close(), ReInit(), and the health checker. Map watch cancellation, generation checks, pending reinitialization, and retry behavior before changing the lifecycle. Done means failed stores are requeued safely without duplicates, superseded watches are cancelled, closed stores are not resurrected, and retries avoid churn.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, grpc
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.