When reindexing, switch to new index once it reached the previous index
- Dominant language
- Scala
- Stars
- 301
- Forks
- 76
- PR merge metrics
- No merged PRs in 30d
Description
When a view gets updated, the following happens:
1. Current view gets stopped
2. Index/namespace associated to it gets wiped
3. New view starts
4. New index/namespace created
5. Start reading from the tags table from the beginning (NoOffset)
This means that the client, for the time it takes to populate the new index, will see no data or less data when querying that index.
A better approach would be:
1. Current view gets paused (no new ingestion)
2. New view starts
3. New index/namespace created (queries will still be redirected to the previous index/namespace)
4. Start reading from the tags table from the beginning (NoOffset)
5. Once the new view reaches the same amount of read events than the previous view (information available from the statistics)
5.1. Delete previous index/namespace
5.2. Run queries against the new index/namespace
Contributor guide
Research direction
The issue names the view update/reindex flow, the tags table, and statistics as starting points; trace those entry points first. Confirm how the previous and new index/namespace are selected during queries. Done means ingestion pauses during reindexing, queries use the previous index until read-event counts match, and the old index/namespace is then removed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- backend, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100