Basekick-Labs / Basekick-Labs/arc
feat(edgesync): schedule spoke replication — today a spoke syncs only when an operator calls POST /spoke-sync/run
- Dominant language
- Go
- Stars
- 677
- Forks
- 53
- Avg merge
- 9h 14m
- Merged PRs (30d)
- 164
Description
## Summary
Edge sync has no scheduler. A spoke replicates only when an operator calls `POST /api/v1/spoke-sync/run`, so unattended edge deployments — the deployment shape the feature exists for — never sync on their own.
This is the "phase 2" the code already refers to by name. `internal/api/edgesync_spoke.go:90-95`:
```go
// Synchronous: the caller gets the pass's outcome rather than a job ID to poll.
// A manual trigger is something an operator watches, and returning immediately
// would mean inventing job tracking for a feature whose automatic form (phase
// 2) will not need it.
```
## Current state
- `Agent.Run` (`internal/edgesync/agent.go:202`) has **exactly one caller**: the HTTP handler at `internal/api/edgesync_spoke.go:108`.
- `grep -rn "Ticker\|interval\|cron\|Schedule" internal/edgesync/*.go` returns **nothing** — the package has no time-driven entry point.
- There is no `edge_sync.spoke.*` interval config key.
So a spoke box in the field with no operator attached accumulates ledger entries indefinitely and delivers nothing.
## What the scheduler has to get right
These are not incidental; each one already has machinery in the tree that a naive ticker would break.
1. **Writer-only execution, checked at every tick.** Per the Cluster Operations Checklist, scheduled jobs gate on `IsPrimaryWriter()` at each tick rather than at `Start()`, so failover and demotion take effect without a restart. Use the `ClusterGate` pattern from `internal/compaction/scheduler.go` as the reference.
2. **No overlapping passes.** `Agent.Run` is a full discover-reconcile-send pass. A tick that fires while the previous pass is still running must skip, not stack. Compaction solves the same problem with `m.cycleRunning`.
3. **Interaction with the compaction defer gate.** When `edge_sync.spoke.defer_compaction_until_synced` is set, compaction waits on delivery (#610). Automatic sync changes the timing of that gate from operator-paced to tick-paced, and `cmd/arc/main.go:1330-1335` installs a fail-closed placeholder gate before the schedulers start specifically because a cron tick can land during a slow startup. A sync scheduler starting in that same window needs the same treatment.
4. **Backoff on a hub that is down.** An edge box with an intermittent uplink is the normal case, not the exceptional one. A fixed interval against an unreachable hub burns the ledger's failure counters; failures need backoff distinct from the success interval.
5. **Bundle mode.** `edge_sync.spoke.bundle.enabled` is a separate path (`POST /export`), and whether it is scheduled on the same timer, a different one, or stays manual is a design decision, not a default.
## Open questions
- One interval, or separate success/failure intervals with backoff?
- Does the scheduler live in `internal/edgesync` (new `scheduler.go`, mirroring `internal/compaction/scheduler.go`) or in `cmd/arc/main.go` wiring?
- Should the manual endpoint stay once the scheduler exists? (Probably yes — it is useful for "sync now before I unplug this box.") If both can run, they need to share the overlap guard in (2).
- Does a scheduled pass emit the same activity-log entries as a manual one, or would that flood the log at a short interval?
- Metrics: a scheduled feature that fails silently is the #802/#810 pattern. Whatever ships needs a last-successful-sync timestamp and a failure counter, wired and verified against a running binary rather than assumed.
## Config shape (proposed, not decided)
```toml
[edge_sync.spoke]
enabled = true
sync_interval = "5m" # new
sync_retry_interval = "30s" # new, on failure
```
Every new key needs `v.SetDefault()` in `internal/config/config.go#setDefaults`, and per CLAUDE.md the binary must be run at least once with each key at a **non-default** value.
## Test plan
- [ ] Scheduler skips a tick while a pass is in flight (no overlapping `Agent.Run`)
- [ ] Gate re-checked every tick: a spoke demoted mid-run stops syncing without a restart
- [ ] Backoff applies on an unreachable hub and resets after a success
- [ ] Interaction with `defer_compaction_until_synced`: compaction is not released by a tick that failed
- [ ] Binary run with `sync_interval` at a non-default value
- [ ] Regression tests must FAIL pre-fix (revert-run-restore)
Related: #611 (`HubIndex.Forget` must be wired before `delete_after_sync` or hub-side retention ships — a blocker on the delete-after-sync half of phase 3), #610 (compaction defer gate).
Contributor guide
Research direction
Start with Agent.Run in internal/edgesync/agent.go and the handler in internal/api/edgesync_spoke.go, then compare internal/compaction/scheduler.go and the startup gate in cmd/arc/main.go:1330-1335. Resolve the listed scheduling, gating, overlap, backoff, bundle, logging, metrics, and configuration questions before implementing. Done means the planned scheduler tests pass, regression tests fail pre-fix, and the binary runs with a non-default sync_interval.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 32/100