cockroachdb / cockroachdb/cockroach

ac: intent resolution can impact foreground latency

Open
#139,188 0 comments 0 reactions 1 assignee Claimed by @angeladietz View on GitHub
A-admission-control branch-master C-bug T-admission-control
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

Intent resolution can cause high raft scheduler latency which results in high SQL latency.

**To Reproduce**

The easiest way to reproduce is to run:

```
PERTURBATION_OVERRIDE=ratioOfMax=.3,numNodes=5,blockSize=4096,disks=1 roachtest run perturbation/full/intents
```

Note that this runs with only 5 nodes and 1 disk, but the same problem can happen with more disks and nodes. Also this sets the ratioOfMax to only 0.3 to reduce the total CPU and IO the foreground load is using.

The 4K block size makes this more pronounced, but it also happens with smaller block sizes.

Specifically this test will cause two outages, but this bug is about the second one. The first one is described in #139187. When the intents are resolved, the raft scheduler latency can spike over 1s resulting in very slow writes. The P99 latency increases to over 500ms.

**Expected behavior**
Intent resolution should not cause as large of an impact on the system. Specifically even though the CPU does not get very high, some requests are still very slow.

**Additional data / screenshots**
![Image](https://github.com/user-attachments/assets/f80501f1-8b38-4ec7-ac40-cc9341b23150)

![Image](https://github.com/user-attachments/assets/71315f58-23d7-4576-a372-03d504b7631b)

![Image](https://github.com/user-attachments/assets/9c8ae4c8-faca-4100-8033-b092ee73cab3)

**Environment:**
- CockroachDB version: master

**Additional context**
In support cases we have seen intent resolution causing system outages and support cases.

We added a setting `kv.transaction.max_intents_and_locks` as part of #135945 to limit the impact of a large number of intents, but this would be better handled by AC.

Jira issue: CRDB-46549

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.