bluewave-labs / bluewave-labs/Checkmate

Data Rollups

Open
#3,853 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
10.8k
Forks
1.2k
Avg merge
1d 4h
Merged PRs (30d)
74

Description

Historical data should be rolled up periodicially for retention

## Considerations

- [ ] Checks are polymorphic, all data needs to be rolled up including PageSpeed and Infra data. Thus the rollup collection should be polymorphic as well, no need for multiple collections
- [ ] What granularity for rollups? Daily is probably sufficient. A daily rollup will serve all date ranges > 1 day. Ranges < 1 day can be query the Check collection directly. Something to consider here though is timezone. If rollups are daily, we'll lose local midnight and will be using UTC midnight. Hourly rollups can resolve this but are more expensive.
- [ ] Scheduling - How often should rollups be done? Once a day is probably sufficient if granularity is daily
- [ ] Backfill - What should happen if rollup data is insufficient for the given date range?
- [ ] Timezone - Server side operations should _all_ be in UTC. Timezone is a client side consideration

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by resolving the open questions about polymorphic data, rollup granularity, scheduling, backfill behavior, and timezone handling; the work is done when a decided design supports retention rollups for all check data and defines how date-range queries use them.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.