Basekick-Labs / Basekick-Labs/arc

arcxrouter: buildEngineSQL lists the entire measurement — no time-based narrowing of the file list

Open
#661 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
677
Forks
53
Avg merge
9h 14m
Merged PRs (30d)
164

Description

## What

`internal/arcxrouter/router.go#buildEngineSQL` builds the engine's file array by `h.Storage.List(ctx, db/measurement/)` over the **whole measurement** (all years), for every routed query. The arcx engine then does its own pruning — bind.rs has exactness-gated min/max statistics pruning and partition-path bucket math (partition.rs), which is data-time-exact — but only over the array it is handed, after the router has paid the full listing cost and materialized every path.

On a high-frequency workload this is the same live-hour cost class that query.file_time_pruning (#660) just fixed on the DuckDB path: a dashboard-shape query (`WHERE time > now() - INTERVAL ...`, the #650-#655 allow-list) routes to arcx and lists ~8K+ live-hour files plus all history, per query.

## Proposed

Reuse the pruner's range extraction when building the engine file list: narrow to hour paths from the query's time range (the same `ExtractTimeRange` the DuckDB path uses), and optionally apply the same filename flush-time pruning for the current hour. The engine's stats pruning stays as the exact second line of defense.

Context: arcx is in development, license-gated + private repo, so exposure is limited today — filed so the cost class is tracked before GA.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Contributor guide

Open the contributing guide

Research direction

Start in internal/arcxrouter/router.go at buildEngineSQL and compare its h.Storage.List(ctx, db/measurement/) call with the DuckDB path's ExtractTimeRange usage. Read the referenced query.file_time_pruning work and the arcx pruning context in bind.rs and partition.rs. Done means routed arcx queries receive only time-relevant hour paths, with current-hour flush-time pruning where applicable, while engine stats pruning remains intact.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
databases, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.