MeshSync: periodic reconciliation to self-heal missed watch events (design)
- Dominant language
- Go
- Stars
- 90
- Forks
- 96
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
Add a periodic reconcile that re-lists watched resources and publishes only genuine drift, to self-heal missed watch events. Today the informer resync period is `0` (pure watch); a watch gap silently drifts MeshSync's snapshot from cluster truth until an external resync is triggered.
## Key decisions
- **A dedicated LIST-vs-Store diff loop, not a non-zero informer resync period** - a client-go resync only replays the local cache and cannot detect events missed entirely (including deletes). The loop diffs a fresh LIST against the informer's `cache.Store`: LIST minus Store = missed add, higher resourceVersion = missed update, **Store minus LIST = missed delete** (reconstruct the tombstone through the existing DeleteFunc path).
- Publishes only drift (reimplements resourceVersion suppression at the reconcile layer) so it never spams Server; steady state = zero publishes.
- User-tunable interval via the MeshSync CRD, **default off**, with an enforced floor; jittered and staggered across GVRs to avoid re-list storms.
## Risks flagged
A partial/failed LIST must abort that GVR's pass for the tick (never diff a truncated list as a mass-delete). Fixes a **pre-existing `h.stores` unguarded-map race** that the loop's concurrent read is the first to expose.
Complementary to JetStream durability (companion blueprint).
## Full design
[docs/design/fd5-periodic-reconciliation.md](https://github.com/meshery/meshsync/blob/master/docs/design/fd5-periodic-reconciliation.md) (added in #576).
Contributor guide
Research direction
Start with docs/design/fd5-periodic-reconciliation.md and review the informer cache.Store and existing DeleteFunc path described in the issue. Trace how the MeshSync CRD would configure the interval, then validate that the design covers drift-only publishing, failed-list aborts, jitter and staggering, and safe concurrent access to h.stores.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- distributed-systems, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100