Basekick-Labs / Basekick-Labs/arc

Continuous queries: event-time windows (watermarking) instead of processing-time boundaries

Open
#523 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Go
Stars
677
Forks
53
Avg merge
9h 14m
Merged PRs (30d)
164

Description

## Summary

CQ window boundaries are based on **processing/wall-clock time** (`endTime = time.Now()`), not the **event time** in the data. Correctness of a given window therefore assumes the data for that window has been ingested by the moment the interval fires. For feeds that are bursty, delayed, or out-of-order (common in financial data), this couples aggregate correctness to ingestion timing rather than to the data's own timestamps.

## Current behavior (code)

- `internal/api/continuous_query.go:464-470` — `endTime = time.Now().UTC()`; the window is `[last_processed_time, now)` on wall-clock. The user's query filters on the data's time column via `{start_time}`/`{end_time}` placeholders, but the **boundary advance is wall-clock**, so events whose event-time falls in a window that has already been closed are missed.

## Proposed direction (larger, architectural)

- Track a **watermark on the data's event-time column** (e.g. max observed event time minus an allowed-lateness bound) and advance CQ windows by event-time watermark rather than wall-clock.
- Lets windows close based on observed data progress, with bounded allowed-lateness, which is the standard streaming-aggregation model.

## Relationship to other issues

- Heavier lift than #522 (lookback) and #521 (idempotency). Those two together cover the near-term financial need (recent late data folded in + no duplicates). Event-time watermarking is the more complete long-term model.
- Likely Enterprise-tier / roadmap, not near-term.

## Evidence
- `internal/api/continuous_query.go:464-470`

Contributor guide

Open the contributing guide

Research direction

Start in internal/api/continuous_query.go:464-470, where endTime is set from time.Now().UTC(), and trace how {start_time}/{end_time} drive the query window and boundary advancement. Research the event-time column and allowed-lateness requirements before proposing the watermark design. Done means CQ windows advance from an event-time watermark and handle delayed or out-of-order data without relying on wall-clock ingestion timing.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.