Basekick-Labs / Basekick-Labs/arc
docs(compaction): hourly MinFiles=10 rationale assumes a 6-minute flush cadence; default is 5 seconds
- Dominant language
- Go
- Stars
- 677
- Forks
- 53
- Avg merge
- 9h 14m
- Merged PRs (30d)
- 164
Description
## Summary
The justification for `MinFiles = 10` in the hourly compaction tier assumes a flush cadence that the current ingest defaults do not produce. The comment says ingestion flushes roughly every 6 minutes; the default age-based flush trigger is **5 seconds** — a ~72x difference.
This is a documentation/threshold-calibration mismatch rather than a correctness bug: `MinFiles` is a *floor*, so a higher real file count still compacts. But the stated rationale no longer describes the system, and the threshold it produced is likely mis-tuned for the low-ingest case.
## The mismatch
`internal/compaction/hourly.go:37-41`:
```go
if cfg.MinFiles == 0 {
// 10 files: ingestion flushes ~every 6 min, so 10 files ≈ 1 hour of data.
// Below this threshold compaction overhead outweighs the read-time savings.
cfg.MinFiles = 10
}
```
Actual ingest flush triggers — whichever fires first, per `database/measurement` buffer:
1. **Row count** — `ingest.max_buffer_size`, default **50000** (`internal/config/config.go:1486`), checked at `internal/ingest/arrow_writer.go:1794` and `:1940`.
2. **Age** — `ingest.max_buffer_age_ms`, default **5000 ms** (`internal/config/config.go:1487`), via `flushAgedBuffers`, `internal/ingest/arrow_writer.go:2463-2480`.
3. **Schema change** — `flushOnSchemaChangeLocked`, `internal/ingest/arrow_writer.go:1769`.
A flush normally writes one Parquet file (`internal/ingest/arrow_writer.go:2621-2691`; multiple only when the flush spans hour boundaries).
So for a measurement ingesting below 50k rows per 5s, the **5-second age trigger dominates → ~720 files/hour/measurement**, not 10. The 6-minute figure would require `max_buffer_age_ms = 360000`.
I could not find any mechanism producing a ~6 minute cadence, so the comment appears to predate a change in the age default.
## Why it matters
- `MinFiles = 10` is reached in ~50 seconds at the default age trigger, so hourly compaction is effectively always eligible — the threshold isn't doing the "is there enough data to be worth it" job the comment describes.
- The real small-files pressure is much larger than the constant implies. This is corroborated elsewhere in the tree: `internal/pruning/file_time_pruning.go:15-18` exists specifically because "the current (live) hour of a high-frequency ingest workload can accumulate thousands of small parquet files (one per buffer flush)."
- Anyone tuning `hourly_min_files` (`internal/config/config.go:1577`) against this comment will reason from a wrong model of flush behavior.
## Suggested fix
1. Correct the comment to reflect the real triggers (`max_buffer_size` 50000 / `max_buffer_age_ms` 5000) and the file counts they imply.
2. Re-derive whether 10 is still the right floor given ~720 files/hour/measurement at low ingest, or whether it should scale with the configured flush cadence.
## Test plan
- [ ] Confirm observed files/hour for a low-rate measurement matches the age-trigger prediction, not the comment
- [ ] Doc-vs-code: no remaining comment in `internal/compaction/` asserts a flush cadence not backed by config
Contributor guide
Research direction
Start with internal/compaction/hourly.go:37-41, then trace the flush defaults in internal/config/config.go:1486-1487 and the triggers in internal/ingest/arrow_writer.go:1769, 1794, 1940, and 2463-2480. Review the file-writing path at :2621-2691 and internal/pruning/file_time_pruning.go:15-18. Done means the rationale matches the configured triggers, the threshold decision is documented, and the test plan confirms the observed file rate.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, performance
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100