cockroachdb / cockroachdb/cockroach
kv: index backfill causes a large impact on foreground latency
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
An index backfill for a table can cause a large impact on foreground latency. In this test the the table the index is created on is otherwise idle, so there isn't any impact of "double writes" changing the behavior.
**To Reproduce**
There are various ways to create a backfill to hit this issue, but the perturbation/*/backfill test frequently hits the issue. The easiest way to reproduce is with the following:
```
PERTURBATION_OVERRIDE=perturbationDuration=30m roachtest run perturbation/full/backfill
```
This will first fill a table for 30 minutes with 1K blocks and then once the system is stable start a low-throughput workload (50%) against a second table.
Once the throughput is stable it starts the backfill and measures the impact.
Running the test with 30 nodes creates an even more pronounced impact.
**Expected behavior**
The throughput and latency of the foreground workload should not be impacted by the backfill. This test specifically splits the table being filled from the table with the index creation to minimize any cross-impacts.
**Additional data / screenshots**
TODO: Add screenshots and data.
**Environment:**
- CockroachDB version: master
**Additional context**
We have had numerous customer case where an index backfill causes a spike in latency. RACv2 (and v1) are intended to address this impact, but it doesn't sufficiently prevent the impact. It is possible that a carefully tuned disk bandwidth limiter might prevent this, however it is quite hard to tune correctly. If it is set too low, the backfill will never complete and if it is set too high it doesn't protect the workload.
Jira issue: CRDB-46576
Epic CRDB-63409
Contributor guide
Assessment
This issue has not been assessed yet.