hashicorp / hashicorp/consul

Add general tombstones or more specific indexes

Open
#2,673 0 comments 0 reactions 0 assignees View on GitHub
theme/internal-cleanup type/enhancement
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

In working through the details of #2671 I realized that there's still quite a bit of room for improvement to blocking queries, even with fine-grained watches. In particular, because we don't have tombstones for anything other than KV, we use max full-table indexes for other types of queries. The fine-grained watches mean we don't wake up unless needed, but then the coarse-grained indexes mean that we don't have good feedback if something actually changed.

We should look at a more general tombstone mechanism or a more general index tracking mechanism that lets us give better-scoped indexes in response to queries. It seems like if we managed the max Raft index in a radix tree alongside the memdb indexes we could do the right thing, even without tombstones. When items are deleted we'd need to propagate the max index from the subtree up to the node above. Need some more thinking and vetting on this, but it seems like a good line to investigate.

Contributor guide

Open the contributing guide

Research direction

Start by reading this issue alongside #2671 and tracing the blocking-query path, fine-grained watches, max full-table indexes, and existing KV tombstones. Investigate whether a general tombstone or Raft-index tracking mechanism with memdb indexes can provide more narrowly scoped query feedback; done requires a vetted design, since no file, test, or concrete acceptance criteria is named.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.