cockroachdb / cockroachdb/cockroach
ac: intent resolution can impact foreground latency
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
Intent resolution can cause high raft scheduler latency which results in high SQL latency.
**To Reproduce**
The easiest way to reproduce is to run:
```
PERTURBATION_OVERRIDE=ratioOfMax=.3,numNodes=5,blockSize=4096,disks=1 roachtest run perturbation/full/intents
```
Note that this runs with only 5 nodes and 1 disk, but the same problem can happen with more disks and nodes. Also this sets the ratioOfMax to only 0.3 to reduce the total CPU and IO the foreground load is using.
The 4K block size makes this more pronounced, but it also happens with smaller block sizes.
Specifically this test will cause two outages, but this bug is about the second one. The first one is described in #139187. When the intents are resolved, the raft scheduler latency can spike over 1s resulting in very slow writes. The P99 latency increases to over 500ms.
**Expected behavior**
Intent resolution should not cause as large of an impact on the system. Specifically even though the CPU does not get very high, some requests are still very slow.
**Additional data / screenshots**



**Environment:**
- CockroachDB version: master
**Additional context**
In support cases we have seen intent resolution causing system outages and support cases.
We added a setting `kv.transaction.max_intents_and_locks` as part of #135945 to limit the impact of a large number of intents, but this would be better handled by AC.
Jira issue: CRDB-46549
Contributor guide
Assessment
This issue has not been assessed yet.