Document how names and IDs can impact blocking query performance
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
As documented in #12343 we generally want (and expect) that `memdb.WatchSet.Watch` will only unblock when a change is made to the item being queried, but in practice that is not always true.
The rules are a little more nuanced:
1. a query that uses [Txn.FirstWatch](https://pkg.go.dev/github.com/hashicorp/go-memdb#Txn.FirstWatch) to query [an index that uses `Unique: true`](https://github.com/hashicorp/go-memdb/blob/82790a6f826ed5de027e24047400f9037fa0cffd/txn.go#L540) and the query finds the requested item, will return a watchCh [for a leaf node](https://github.com/hashicorp/go-immutable-radix/blob/12e3e257ae80399bf1fb037f6e5bac87e4648875/node.go#L112-L114). In this case the `watchCh` should only be closed when the query result actually changes.
2. a query that uses `Txn.FirstWatch` where either the index is not unique, or no value was found, will return a `watchCh` using [Iterator.SeekPrefixWatch](https://github.com/hashicorp/go-immutable-radix/blob/49d1d02/iter.go#L16-L51). The `watchCh` will be from an edge node or the root node, which means it may be modified by inserts or deletes of other items.
3. any query that uses `Txn.Get` will always return a `watchCh` using `Iterator.SeekPrefixWatch`, from an edge node or the root node.
As [this test case demonstrates](https://github.com/hashicorp/consul/compare/dnephin/prefix-overlap-watches), when `Iterator.SeekPrefixWatch` is used to return a `watchCh` from an edge node, inserts or deletes of unrelated items can cause the `WatchSet.Watch` to fire. In these cases (referring back to the rules described in #12343) we have to rely on the `responseMeta.Index` to indicate if the results have changed or not. Often that index comes from the "last modified index of the table", which means it will also have changed, and the blocking query will return before the timeout, even though the results have not changed.
Some tables (Ex: `services`) maintain a per-service "last modified" index. This per-service index should safeguard against the `WatchSet` unblocking, because `Server.blockingQuery` will recognize there is no change (by comparing indexes), and repeat the query without returning an RPC response.
However most other tables do not use per-item tracking of modified index, which means those queries can return more frequently. How frequently depends on the names being used and how often items are inserted or deleted. There are currently no log lines that can help indicate how frequently this happens, but #11947 proposes a change that would make it easy to observe this happening in production.
Every insert or delete does not necessarily cause the `watchCh` to be closed. It only happens when the insert or delete causes the tree to be restructured. Since these indexes are based on the names and IDs from the user, the user has some control over the impact of these extra queries.
We should probably document this somehow, or otherwise look to safe-guard other queries using the per-item last modified index tracking.
Contributor guide
Research direction
Start with #12343 and the referenced `txn.go`, `iter.go`, and `Server.blockingQuery` behavior, then review the linked prefix-overlap watch test case. Document when `FirstWatch` and `Get` can unblock for unrelated changes, how names and IDs affect this, and where per-item modified indexes provide protection. Done means the relevant user-facing documentation and any recommended safeguards are identified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend-api-design, documentation, performance
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 32/100