Basekick-Labs / Basekick-Labs/arc
Separate compaction schedulers cause daily tier starvation
- Dominant language
- Go
- Stars
- 677
- Forks
- 53
- Avg merge
- 9h 14m
- Merged PRs (30d)
- 164
Description
## Problem
Hourly and daily compaction use separate `Scheduler` instances sharing one `Manager` with a single `cycleRunning` atomic (`cmd/arc/main.go:611-663`). When hourly holds `cycleRunning`, daily gets `ErrCycleAlreadyRunning` and silently skips.
Daily fires once/day at 3:00 AM. Hourly fires 24x/day. If any hourly cycle overlaps with daily's window, daily is starved.
## Confirmed Issues
| Issue | Test Result |
|-------|------------|
| **Daily starvation** | Blocked ALL 5 simulated days — never ran |
| **Hierarchy bypass** | 42 raw files skipped by hourly (below MinFiles), processed directly by daily |
| **Re-download waste** | Daily re-downloaded 10 `_compacted.parquet` files hourly just uploaded |
| **Lock key mismatch** | Hourly locks `db/meas/YYYY/MM/DD/HH`, daily locks `db/meas/YYYY/MM/DD` — no mutual exclusion |
| **File overlap** | 15 files targeted by both tiers simultaneously |
## Fix
Replace two schedulers with one. `runCycleInternal()` already processes tiers sequentially — just call them together:
```go
// BEFORE
hourlyScheduler = NewScheduler(&SchedulerConfig{TierNames: []string{"hourly"}, Schedule: "5 * * * *"})
dailyScheduler = NewScheduler(&SchedulerConfig{TierNames: []string{"daily"}, Schedule: "0 3 * * *"})
// AFTER
compactionScheduler = NewScheduler(&SchedulerConfig{
TierNames: []string{"hourly", "daily"},
Schedule: "5 * * * *",
})
```
Contributor guide
Assessment
This issue has not been assessed yet.