bluewave-labs / bluewave-labs/Checkmate
Data Rollups
- Dominant language
- TypeScript
- Stars
- 10.8k
- Forks
- 1.2k
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 74
Description
Historical data should be rolled up periodicially for retention
## Considerations
- [ ] Checks are polymorphic, all data needs to be rolled up including PageSpeed and Infra data. Thus the rollup collection should be polymorphic as well, no need for multiple collections
- [ ] What granularity for rollups? Daily is probably sufficient. A daily rollup will serve all date ranges > 1 day. Ranges < 1 day can be query the Check collection directly. Something to consider here though is timezone. If rollups are daily, we'll lose local midnight and will be using UTC midnight. Hourly rollups can resolve this but are more expensive.
- [ ] Scheduling - How often should rollups be done? Once a day is probably sufficient if granularity is daily
- [ ] Backfill - What should happen if rollup data is insufficient for the given date range?
- [ ] Timezone - Server side operations should _all_ be in UTC. Timezone is a client side consideration
Contributor guide
Research direction
No files, tests, or entry points are named. Start by resolving the open questions about polymorphic data, rollup granularity, scheduling, backfill behavior, and timezone handling; the work is done when a decided design supports retention rollups for all check data and defines how date-range queries use them.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100