cockroachdb / cockroachdb/cockroach

concurrency: consider wait metrics by request category

Open
#159,685 1 comment 0 reactions 0 assignees View on GitHub
A-kv-transactions C-enhancement T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

The latency of BatchRequests include time spent waiting for various forms of contention, most notably:

- Latch wait times
- Lock wait times

For the former we currently have a standard histogram metric and for the latter we record a max and average metric.

While we expect latch wait times to be relatively low even in the face of contention, lock wait times can be arbitrarily large, historically making metrics here less useful.

However, as we work on non-blocking, non-locking reads, we expect some categories of requests (namely read-only batches) to experience much lower lock wait times.

We should extend our metrics to get a signal about whether we are actually achieving that goal in production installations. One way to do that would be to have a histogram on lock and latch wait times that were broken down by the type of batch (say: "admin", "read-only", "read-write"). We would then be able to track "read-only" wait times in a cluster without detailed knowledge of its workload.

Jira issue: CRDB-57985

Contributor guide

Open the contributing guide

Research direction

Start by locating the BatchRequests implementation and the existing latch and lock wait metrics described in the issue. Determine how request categories such as admin, read-only, and read-write are represented, then verify that the resulting wait-time signals distinguish those categories in production-oriented metrics.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.