Incrementally update roll-ups instead of refreshing materialized views
- Dominant language
- Scala
- Stars
- 81
- Forks
- 26
- PR merge metrics
- No merged PRs in 30d
Description
Using materialized views allows us to write the raw aggregations (per-changeset) and roll them up periodically from nothing.
There are a couple substantial downsides from this approach that become more apparent at scale:
* roll-ups aren't fully up-to-date (materialized views needs to be refreshed, at which point it is almost immediate out-of-date)
* refreshing materialized views require a full scan of the source table(s) and may take minutes to hours on larger tables (Missing Maps leaderboards use this approach and can take several hours to update)
Refreshing views on a loop isn't going to work out long-term (and already isn't working for MM).
Contributor guide
No contributing guide indexed for this repository
Research direction
No files, tests, or entry points are identified in the issue. Start by tracing where raw per-changeset aggregations are written and where materialized views are refreshed, then define the incremental roll-up approach and verify that updates no longer require a full source-table scan while keeping roll-ups current.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100