Basekick-Labs / Basekick-Labs/arc

Separate compaction schedulers cause daily tier starvation

Open
#220 1 comment 0 reactions 0 assignees View on GitHub
claude
Dominant language
Go
Stars
677
Forks
53
Avg merge
9h 14m
Merged PRs (30d)
164

Description

## Problem

Hourly and daily compaction use separate `Scheduler` instances sharing one `Manager` with a single `cycleRunning` atomic (`cmd/arc/main.go:611-663`). When hourly holds `cycleRunning`, daily gets `ErrCycleAlreadyRunning` and silently skips.

Daily fires once/day at 3:00 AM. Hourly fires 24x/day. If any hourly cycle overlaps with daily's window, daily is starved.

## Confirmed Issues

| Issue | Test Result |
|-------|------------|
| **Daily starvation** | Blocked ALL 5 simulated days — never ran |
| **Hierarchy bypass** | 42 raw files skipped by hourly (below MinFiles), processed directly by daily |
| **Re-download waste** | Daily re-downloaded 10 `_compacted.parquet` files hourly just uploaded |
| **Lock key mismatch** | Hourly locks `db/meas/YYYY/MM/DD/HH`, daily locks `db/meas/YYYY/MM/DD` — no mutual exclusion |
| **File overlap** | 15 files targeted by both tiers simultaneously |

## Fix

Replace two schedulers with one. `runCycleInternal()` already processes tiers sequentially — just call them together:

```go
// BEFORE
hourlyScheduler = NewScheduler(&SchedulerConfig{TierNames: []string{"hourly"}, Schedule: "5 * * * *"})
dailyScheduler = NewScheduler(&SchedulerConfig{TierNames: []string{"daily"}, Schedule: "0 3 * * *"})

// AFTER
compactionScheduler = NewScheduler(&SchedulerConfig{
TierNames: []string{"hourly", "daily"},
Schedule: "5 * * * *",
})
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.