cockroachdb / cockroachdb/cockroach

roachtest: create a roachtest demonstrating read performance during point tombstone buildup

Open
#129,660 1 comment 0 reactions 0 assignees View on GitHub
A-storage C-enhancement T-storage
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

As described in https://github.com/cockroachdb/pebble/issues/918, certain workloads are prone to buildup of point tombstones at the storage layer, which can significantly degrade read latency. This was addressed in https://github.com/cockroachdb/pebble/pull/3790, but we should have a roachtest to demonstrate the performance improvement at the CockroachDB level.

Workloads involving a queue are especially susceptible to this issue. The section titled "CockroachDB Workload" in the comment [here](https://github.com/cockroachdb/pebble/pull/3790#issuecomment-2251439492) describes a queue-based workload where we saw read performance slowly worsen over time as tombstones build up. This could be adapted directly into a roachtest, but it may also be worth investigating other workload types since average latency still only degrades from 1ms -> 7ms in that case before the tombstones are compacted away by previously existing heuristics. We have seen cases where reads can take multiple seconds, e.g. in the liveness range (https://github.com/cockroachdb/cockroach/issues/128968).

Jira issue: CRDB-41649

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.