cockroachdb / cockroachdb/cockroach
kv: consider bumping the timestamp cache on write intent errors for reads that go through pessimistic evaluation
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Is your feature request related to a problem? Please describe.**
Currently, readers that run into `WriteIntentErrors` wait for the intent to be resolved without bumping the timestamp cache. This means other concurrent writers are able to freely write intents on the read's read span. These will only be discovered once the read tries to re-evaluate after resolving intents its first scan encountered.
The current scheme described above increases concurrency for writers. It does so in 2 ways:
1. Writers are able to proceed without bumping their timestamp above the read's timestamp. This reduces the possibility of the writing transaction needing to refresh its read set, which in-turn increases its chances to commit.
2. Reads that hold latches during their evaluation (read: small limit scans that go through optimistic evaluation) only bump their timestamp cache post evaluation and only for the keys that were actually read. This reduces false contention between reads that declare large read spans but only scan a few keys and writers writing to keys that were not read.
However, under the current scheme, the reader can be starved out by a steady stream of writers writing "in front of the reader" right after the reader has collected a set of intents from its scan. Note that this hazard is rare, because we don't expect writers to indefinitely perform writes below the read's timestamp. However, we have seen something like this happen when large reads and writes interact -- specifically, an `ExportRequest` and a large `DELETE` statement (https://github.com/cockroachdb/cockroach/issues/101683).
One way to prevent this starvation is to bump the timestamp cache over the read's declared read set. This would mean concurrent writers will not be able to write intents that will be discovered by the read on re-evaluation(s). We probably don't want to do this for all reads though, because doing so would reduce concurrency between readers and writers. One suggestion would be to do it for reads that go through `PessimisticEval` (we already use `PessimisticEval` as a proxy for whether or not to drop latches early, which presents a similar tradeoff). Alternatively, we could use a different heuristic for what "large" means.
cc @nvanbenschoten
Jira issue: CRDB-27434
Contributor guide
Research direction
Start by tracing the read paths involving WriteIntentErrors and PessimisticEval, then review how the timestamp cache is updated after evaluation. Use the ExportRequest and large DELETE interaction in issue #101683 as the motivating case. Done requires agreeing on a heuristic and implementing it with coverage for the starvation scenario.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100