Basekick-Labs / Basekick-Labs/arc
Track experimental file-level time pruning toward stable + default-on in 27.01.1
- Dominant language
- Go
- Stars
- 677
- Forks
- 53
- Avg merge
- 9h 14m
- Merged PRs (30d)
- 164
Description
## What
`query.file_time_pruning` ships in **26.09.2 (October) as experimental, opt-in** (default off). This issue tracks its soak in our dev environment (the Apex agent workload: one-second ingest, ~8K files/live hour, continuous dashboard queries) toward promotion.
## Measured at introduction (2026-08-28)
- 5-minute dashboard query over a 7,893-file live hour: **340ms → 29ms (11.7×)**; 60s window 12ms
- Side effect: arc RSS on the same workload dropped from ~450MB to ~90MB (DuckDB file/object caches stop inflating with live-hour file count)
- Ships alongside the unquoted-`INTERVAL` pruning fix (DuckDB-native syntax previously got zero partition pruning)
## Check-in (late September 2026)
- [ ] Multi-week soak on the dev environment: no wrong-results reports, no missing live rows on dashboards
- [ ] RSS and query-latency curves stayed flat across many hour rollovers (monitor data)
- [ ] No regressions attributable to the always-on unquoted-INTERVAL extraction
- [ ] Revisit the same-hour future-clock-skew contract: is the 300s default margin right?
- [ ] Field feedback from any 26.09.2 opt-in users
## Promotion criteria → 27.01.1
If all of the above hold: flip `query.file_time_pruning` **default to true** and drop the experimental label in 27.01.1. If anything wobbles, it stays opt-in and this issue documents why.
Related follow-ups filed separately: arcx router whole-measurement listing (same live-hour cost, bypasses pruner); cold-tier path skips all pruning; absolute-time patterns matching escaped datetimes inside literals (pre-existing B2-class).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Contributor guide
Research direction
Start by locating the implementation and default registration for `query.file_time_pruning`, then review the Apex agent workload described in the issue. Run or inspect the multi-week dev-environment soak and monitor wrong results, missing live rows, RSS, query latency, and hour rollovers. Done means the promotion criteria are met and the default can be flipped in 27.01.1, or the issue records why it remains opt-in.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sql
- Domain
- databases, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100