MeshSync: rate-limited work queue for backpressure (design)
- Dominant language
- Go
- Stars
- 90
- Forks
- 96
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
Insert a bounded, rate-limited work queue + converter between informer event delivery and the broker publish, so a slow/unavailable NATS broker cannot stall the informer's single event-processing goroutine, and mass-rollout bursts are smoothed. Today `internal/pipeline/handlers.go` publishes inline/synchronously with no queue, no backpressure, and no retry.
## Key decisions
- Use `k8s.io/client-go/util/workqueue` (already a dependency) with a "latest-value map keyed by object identity" for coalescing: the queue carries keys, MeshSync's converted-object cache carries values. Wrap at the `output.Writer` seam so handler code is untouched.
- **Asymmetric overflow policy** (the load-bearing decision): DELETE is never dropped (bounded-block); ADD/UPDATE bounded-drop-with-metric (safe because resync re-delivers). Hard drain-timeout on shutdown so the stall isn't merely moved to teardown.
- The metrics interface is the seam a future `/metrics` endpoint hangs on.
## Scope / phasing
PR1 QueuingWriter + tests (unwired); PR2 wire into pipeline + runtime-verify under load; PR3 Operator CRD knobs (workerCount/queueSize). ~1.5-2 wks.
**Must-verify before merge:** confirm the informer resync actually re-fires AddFunc for already-known objects - the ADD/UPDATE drop-safety argument depends on it.
## Full design
[docs/design/fd2-rate-limited-work-queue.md](https://github.com/meshery/meshsync/blob/master/docs/design/fd2-rate-limited-work-queue.md) (added in #576).
Contributor guide
Research direction
Read docs/design/fd2-rate-limited-work-queue.md, then inspect internal/pipeline/handlers.go and the output.Writer seam. Start with the un wired PR1 QueuingWriter and its tests, and verify whether informer resync re-fires AddFunc; completion covers the stated phased work and load verification.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- backend, distributed-systems, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100