Basekick-Labs / Basekick-Labs/arc

feat(edgesync): schedule spoke replication — today a spoke syncs only when an operator calls POST /spoke-sync/run

Open
#828 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Go
Stars
677
Forks
53
Avg merge
9h 14m
Merged PRs (30d)
164

Description

## Summary

Edge sync has no scheduler. A spoke replicates only when an operator calls `POST /api/v1/spoke-sync/run`, so unattended edge deployments — the deployment shape the feature exists for — never sync on their own.

This is the "phase 2" the code already refers to by name. `internal/api/edgesync_spoke.go:90-95`:

```go
// Synchronous: the caller gets the pass's outcome rather than a job ID to poll.
// A manual trigger is something an operator watches, and returning immediately
// would mean inventing job tracking for a feature whose automatic form (phase
// 2) will not need it.
```

## Current state

- `Agent.Run` (`internal/edgesync/agent.go:202`) has **exactly one caller**: the HTTP handler at `internal/api/edgesync_spoke.go:108`.
- `grep -rn "Ticker\|interval\|cron\|Schedule" internal/edgesync/*.go` returns **nothing** — the package has no time-driven entry point.
- There is no `edge_sync.spoke.*` interval config key.

So a spoke box in the field with no operator attached accumulates ledger entries indefinitely and delivers nothing.

## What the scheduler has to get right

These are not incidental; each one already has machinery in the tree that a naive ticker would break.

1. **Writer-only execution, checked at every tick.** Per the Cluster Operations Checklist, scheduled jobs gate on `IsPrimaryWriter()` at each tick rather than at `Start()`, so failover and demotion take effect without a restart. Use the `ClusterGate` pattern from `internal/compaction/scheduler.go` as the reference.

2. **No overlapping passes.** `Agent.Run` is a full discover-reconcile-send pass. A tick that fires while the previous pass is still running must skip, not stack. Compaction solves the same problem with `m.cycleRunning`.

3. **Interaction with the compaction defer gate.** When `edge_sync.spoke.defer_compaction_until_synced` is set, compaction waits on delivery (#610). Automatic sync changes the timing of that gate from operator-paced to tick-paced, and `cmd/arc/main.go:1330-1335` installs a fail-closed placeholder gate before the schedulers start specifically because a cron tick can land during a slow startup. A sync scheduler starting in that same window needs the same treatment.

4. **Backoff on a hub that is down.** An edge box with an intermittent uplink is the normal case, not the exceptional one. A fixed interval against an unreachable hub burns the ledger's failure counters; failures need backoff distinct from the success interval.

5. **Bundle mode.** `edge_sync.spoke.bundle.enabled` is a separate path (`POST /export`), and whether it is scheduled on the same timer, a different one, or stays manual is a design decision, not a default.

## Open questions

- One interval, or separate success/failure intervals with backoff?
- Does the scheduler live in `internal/edgesync` (new `scheduler.go`, mirroring `internal/compaction/scheduler.go`) or in `cmd/arc/main.go` wiring?
- Should the manual endpoint stay once the scheduler exists? (Probably yes — it is useful for "sync now before I unplug this box.") If both can run, they need to share the overlap guard in (2).
- Does a scheduled pass emit the same activity-log entries as a manual one, or would that flood the log at a short interval?
- Metrics: a scheduled feature that fails silently is the #802/#810 pattern. Whatever ships needs a last-successful-sync timestamp and a failure counter, wired and verified against a running binary rather than assumed.

## Config shape (proposed, not decided)

```toml
[edge_sync.spoke]
enabled = true
sync_interval = "5m" # new
sync_retry_interval = "30s" # new, on failure
```

Every new key needs `v.SetDefault()` in `internal/config/config.go#setDefaults`, and per CLAUDE.md the binary must be run at least once with each key at a **non-default** value.

## Test plan

- [ ] Scheduler skips a tick while a pass is in flight (no overlapping `Agent.Run`)
- [ ] Gate re-checked every tick: a spoke demoted mid-run stops syncing without a restart
- [ ] Backoff applies on an unreachable hub and resets after a success
- [ ] Interaction with `defer_compaction_until_synced`: compaction is not released by a tick that failed
- [ ] Binary run with `sync_interval` at a non-default value
- [ ] Regression tests must FAIL pre-fix (revert-run-restore)

Related: #611 (`HubIndex.Forget` must be wired before `delete_after_sync` or hub-side retention ships — a blocker on the delete-after-sync half of phase 3), #610 (compaction defer gate).

Contributor guide

Open the contributing guide

Research direction

Start with Agent.Run in internal/edgesync/agent.go and the handler in internal/api/edgesync_spoke.go, then compare internal/compaction/scheduler.go and the startup gate in cmd/arc/main.go:1330-1335. Resolve the listed scheduling, gating, overlap, backoff, bundle, logging, metrics, and configuration questions before implementing. Done means the planned scheduler tests pass, regression tests fail pre-fix, and the binary runs with a non-default sync_interval.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.