Basekick-Labs / Basekick-Labs/arc

Track experimental file-level time pruning toward stable + default-on in 27.01.1

Open
#659 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
677
Forks
53
Avg merge
9h 14m
Merged PRs (30d)
164

Description

## What

`query.file_time_pruning` ships in **26.09.2 (October) as experimental, opt-in** (default off). This issue tracks its soak in our dev environment (the Apex agent workload: one-second ingest, ~8K files/live hour, continuous dashboard queries) toward promotion.

## Measured at introduction (2026-08-28)

- 5-minute dashboard query over a 7,893-file live hour: **340ms → 29ms (11.7×)**; 60s window 12ms
- Side effect: arc RSS on the same workload dropped from ~450MB to ~90MB (DuckDB file/object caches stop inflating with live-hour file count)
- Ships alongside the unquoted-`INTERVAL` pruning fix (DuckDB-native syntax previously got zero partition pruning)

## Check-in (late September 2026)

- [ ] Multi-week soak on the dev environment: no wrong-results reports, no missing live rows on dashboards
- [ ] RSS and query-latency curves stayed flat across many hour rollovers (monitor data)
- [ ] No regressions attributable to the always-on unquoted-INTERVAL extraction
- [ ] Revisit the same-hour future-clock-skew contract: is the 300s default margin right?
- [ ] Field feedback from any 26.09.2 opt-in users

## Promotion criteria → 27.01.1

If all of the above hold: flip `query.file_time_pruning` **default to true** and drop the experimental label in 27.01.1. If anything wobbles, it stays opt-in and this issue documents why.

Related follow-ups filed separately: arcx router whole-measurement listing (same live-hour cost, bypasses pruner); cold-tier path skips all pruning; absolute-time patterns matching escaped datetimes inside literals (pre-existing B2-class).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Contributor guide

Open the contributing guide

Research direction

Start by locating the implementation and default registration for `query.file_time_pruning`, then review the Apex agent workload described in the issue. Run or inspect the multi-week dev-environment soak and monitor wrong results, missing live rows, RSS, query latency, and hour rollovers. Done means the promotion criteria are met and the default can be flipped in 27.01.1, or the issue records why it remains opt-in.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
databases, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.