Basekick-Labs / Basekick-Labs/arc

docs(compaction): hourly MinFiles=10 rationale assumes a 6-minute flush cadence; default is 5 seconds

Open
#793 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
677
Forks
53
Avg merge
9h 14m
Merged PRs (30d)
164

Description

## Summary

The justification for `MinFiles = 10` in the hourly compaction tier assumes a flush cadence that the current ingest defaults do not produce. The comment says ingestion flushes roughly every 6 minutes; the default age-based flush trigger is **5 seconds** — a ~72x difference.

This is a documentation/threshold-calibration mismatch rather than a correctness bug: `MinFiles` is a *floor*, so a higher real file count still compacts. But the stated rationale no longer describes the system, and the threshold it produced is likely mis-tuned for the low-ingest case.

## The mismatch

`internal/compaction/hourly.go:37-41`:
```go
if cfg.MinFiles == 0 {
// 10 files: ingestion flushes ~every 6 min, so 10 files ≈ 1 hour of data.
// Below this threshold compaction overhead outweighs the read-time savings.
cfg.MinFiles = 10
}
```

Actual ingest flush triggers — whichever fires first, per `database/measurement` buffer:

1. **Row count** — `ingest.max_buffer_size`, default **50000** (`internal/config/config.go:1486`), checked at `internal/ingest/arrow_writer.go:1794` and `:1940`.
2. **Age** — `ingest.max_buffer_age_ms`, default **5000 ms** (`internal/config/config.go:1487`), via `flushAgedBuffers`, `internal/ingest/arrow_writer.go:2463-2480`.
3. **Schema change** — `flushOnSchemaChangeLocked`, `internal/ingest/arrow_writer.go:1769`.

A flush normally writes one Parquet file (`internal/ingest/arrow_writer.go:2621-2691`; multiple only when the flush spans hour boundaries).

So for a measurement ingesting below 50k rows per 5s, the **5-second age trigger dominates → ~720 files/hour/measurement**, not 10. The 6-minute figure would require `max_buffer_age_ms = 360000`.

I could not find any mechanism producing a ~6 minute cadence, so the comment appears to predate a change in the age default.

## Why it matters

- `MinFiles = 10` is reached in ~50 seconds at the default age trigger, so hourly compaction is effectively always eligible — the threshold isn't doing the "is there enough data to be worth it" job the comment describes.
- The real small-files pressure is much larger than the constant implies. This is corroborated elsewhere in the tree: `internal/pruning/file_time_pruning.go:15-18` exists specifically because "the current (live) hour of a high-frequency ingest workload can accumulate thousands of small parquet files (one per buffer flush)."
- Anyone tuning `hourly_min_files` (`internal/config/config.go:1577`) against this comment will reason from a wrong model of flush behavior.

## Suggested fix

1. Correct the comment to reflect the real triggers (`max_buffer_size` 50000 / `max_buffer_age_ms` 5000) and the file counts they imply.
2. Re-derive whether 10 is still the right floor given ~720 files/hour/measurement at low ingest, or whether it should scale with the configured flush cadence.

## Test plan

- [ ] Confirm observed files/hour for a low-rate measurement matches the age-trigger prediction, not the comment
- [ ] Doc-vs-code: no remaining comment in `internal/compaction/` asserts a flush cadence not backed by config

Contributor guide

Open the contributing guide

Research direction

Start with internal/compaction/hourly.go:37-41, then trace the flush defaults in internal/config/config.go:1486-1487 and the triggers in internal/ingest/arrow_writer.go:1769, 1794, 1940, and 2463-2480. Review the file-writing path at :2621-2691 and internal/pruning/file_time_pruning.go:15-18. Done means the rationale matches the configured triggers, the threshold decision is documented, and the test plan confirms the observed file rate.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, performance
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.