meshery / meshery/meshsync

MeshSync: rate-limited work queue for backpressure (design)

Open
#578 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
90
Forks
96
PR merge metrics
No merged PRs in 30d

Description

## Summary
Insert a bounded, rate-limited work queue + converter between informer event delivery and the broker publish, so a slow/unavailable NATS broker cannot stall the informer's single event-processing goroutine, and mass-rollout bursts are smoothed. Today `internal/pipeline/handlers.go` publishes inline/synchronously with no queue, no backpressure, and no retry.

## Key decisions
- Use `k8s.io/client-go/util/workqueue` (already a dependency) with a "latest-value map keyed by object identity" for coalescing: the queue carries keys, MeshSync's converted-object cache carries values. Wrap at the `output.Writer` seam so handler code is untouched.
- **Asymmetric overflow policy** (the load-bearing decision): DELETE is never dropped (bounded-block); ADD/UPDATE bounded-drop-with-metric (safe because resync re-delivers). Hard drain-timeout on shutdown so the stall isn't merely moved to teardown.
- The metrics interface is the seam a future `/metrics` endpoint hangs on.

## Scope / phasing
PR1 QueuingWriter + tests (unwired); PR2 wire into pipeline + runtime-verify under load; PR3 Operator CRD knobs (workerCount/queueSize). ~1.5-2 wks.

**Must-verify before merge:** confirm the informer resync actually re-fires AddFunc for already-known objects - the ADD/UPDATE drop-safety argument depends on it.

## Full design
[docs/design/fd2-rate-limited-work-queue.md](https://github.com/meshery/meshsync/blob/master/docs/design/fd2-rate-limited-work-queue.md) (added in #576).

Contributor guide

Open the contributing guide

Research direction

Read docs/design/fd2-rate-limited-work-queue.md, then inspect internal/pipeline/handlers.go and the output.Writer seam. Start with the un wired PR1 QueuingWriter and its tests, and verify whether informer resync re-fires AddFunc; completion covers the stated phased work and load verification.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend, distributed-systems, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.