Basekick-Labs / Basekick-Labs/arc

Continuous queries: handle late / out-of-order data (lookback window + idempotent upsert)

Open
#522 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Go
Stars
677
Forks
53
Avg merge
9h 14m
Merged PRs (30d)
164

Description

## Summary

Continuous queries compute each window **exactly once** over `[last_processed_time, now)` and then advance the watermark past it, with **no lookback/grace window**. Data that arrives after its window has already been processed (late ticks, corrections, backfills, out-of-order events) is **silently dropped from that window's aggregate**. There is no way to recover it short of manually re-running.

This is acceptable for clean, in-order infra metrics, but it is a blocker for **financial roll-ups** (OHLC bars, VWAP, PnL/position aggregates) where late corrections and out-of-order events are normal and materially change results. Raised by a prospect evaluating Arc for financial data.

## Current behavior (code)

- `internal/api/continuous_query.go:464-470` — window start = previous `last_processed_time`; end = `time.Now()` (wall-clock).
- No lookback/grace: `startTime` is exactly the prior end (`:468/:586` only set a default on first run).
- Watermark advances to `endTime` after each run (`:504`), so earlier windows are never revisited.

## Proposed feature

1. **Configurable lookback / grace window** per CQ: each run reprocesses the last N intervals (or a duration) so recent late data is folded in. Example knob: `reprocess_window` / `late_data_grace`.
2. **Idempotent output (depends on #521):** reprocessing a window must REPLACE that window's prior aggregate rows, not append. This requires the idempotency fix in #521 (replace/upsert by window+tags). Without it, lookback just multiplies duplicates.
3. (Optional, later) **Event-time boundary** instead of wall-clock `now()` — see separate event-time issue.

## Acceptance criteria

- A CQ with a lookback of K intervals, given a row that lands one interval late, produces an aggregate for the correct window on the next run (not dropped).
- Re-running an already-computed window overwrites its output (no duplicates), enabled by #521.
- Default behavior (no lookback configured) is unchanged for existing users.

## Notes

- Strong candidate for an Enterprise feature — a paying financial customer needs it.
- Blocked on / coupled with #521 (idempotent CQ output).

Contributor guide

Open the contributing guide

Research direction

Start in internal/api/continuous_query.go at lines 464-470 and 504, then inspect the related watermark handling around lines 586. Review issue #521 before designing the configurable lookback, since reprocessing must replace prior window aggregates rather than append duplicates. Done means late rows reach the correct window, repeated runs produce no duplicates, and the default behavior remains unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.