Basekick-Labs / Basekick-Labs/arc
Continuous queries: event-time windows (watermarking) instead of processing-time boundaries
- Dominant language
- Go
- Stars
- 677
- Forks
- 53
- Avg merge
- 9h 14m
- Merged PRs (30d)
- 164
Description
## Summary
CQ window boundaries are based on **processing/wall-clock time** (`endTime = time.Now()`), not the **event time** in the data. Correctness of a given window therefore assumes the data for that window has been ingested by the moment the interval fires. For feeds that are bursty, delayed, or out-of-order (common in financial data), this couples aggregate correctness to ingestion timing rather than to the data's own timestamps.
## Current behavior (code)
- `internal/api/continuous_query.go:464-470` — `endTime = time.Now().UTC()`; the window is `[last_processed_time, now)` on wall-clock. The user's query filters on the data's time column via `{start_time}`/`{end_time}` placeholders, but the **boundary advance is wall-clock**, so events whose event-time falls in a window that has already been closed are missed.
## Proposed direction (larger, architectural)
- Track a **watermark on the data's event-time column** (e.g. max observed event time minus an allowed-lateness bound) and advance CQ windows by event-time watermark rather than wall-clock.
- Lets windows close based on observed data progress, with bounded allowed-lateness, which is the standard streaming-aggregation model.
## Relationship to other issues
- Heavier lift than #522 (lookback) and #521 (idempotency). Those two together cover the near-term financial need (recent late data folded in + no duplicates). Event-time watermarking is the more complete long-term model.
- Likely Enterprise-tier / roadmap, not near-term.
## Evidence
- `internal/api/continuous_query.go:464-470`
Contributor guide
Research direction
Start in internal/api/continuous_query.go:464-470, where endTime is set from time.Now().UTC(), and trace how {start_time}/{end_time} drive the query window and boundary advancement. Research the event-time column and allowed-lateness requirements before proposing the watermark design. Done means CQ windows advance from an event-time watermark and handle delayed or out-of-order data without relying on wall-clock ingestion timing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100