cockroachdb / cockroachdb/cockroach

kv: index backfill causes a large impact on foreground latency

Open
#139,262 6 comments 0 reactions 1 assignee Claimed by @angeladietz View on GitHub
A-admission-control A-replication-admission-control-v2 branch-master C-bug O-perturbation T-admission-control
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

An index backfill for a table can cause a large impact on foreground latency. In this test the the table the index is created on is otherwise idle, so there isn't any impact of "double writes" changing the behavior.

**To Reproduce**

There are various ways to create a backfill to hit this issue, but the perturbation/*/backfill test frequently hits the issue. The easiest way to reproduce is with the following:

```
PERTURBATION_OVERRIDE=perturbationDuration=30m roachtest run perturbation/full/backfill
```

This will first fill a table for 30 minutes with 1K blocks and then once the system is stable start a low-throughput workload (50%) against a second table.

Once the throughput is stable it starts the backfill and measures the impact.

Running the test with 30 nodes creates an even more pronounced impact.

**Expected behavior**
The throughput and latency of the foreground workload should not be impacted by the backfill. This test specifically splits the table being filled from the table with the index creation to minimize any cross-impacts.

**Additional data / screenshots**
TODO: Add screenshots and data.

**Environment:**
- CockroachDB version: master

**Additional context**
We have had numerous customer case where an index backfill causes a spike in latency. RACv2 (and v1) are intended to address this impact, but it doesn't sufficiently prevent the impact. It is possible that a carefully tuned disk bandwidth limiter might prevent this, however it is quite hard to tune correctly. If it is set too low, the backfill will never complete and if it is set too high it doesn't protect the workload.

Jira issue: CRDB-46576

Epic CRDB-63409

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.