cockroachdb / cockroachdb/cockroach

sql,kv: sqlinstances based planning causes high goroutine scheduling latency because of bursty system.sqlliveness traffic

Open
#147,755 11 comments 0 reactions 1 assignee Claimed by @cthumuluru-crdb View on GitHub
branch-master C-bug O-25.2.1-scale-testing O-25.4-scale-testing O-testcluster P-2 T-db-server
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

In a large scale test, we noted that the leaseholder of the range holding the `system.sqlliveness` table was consistently observing periods of high goroutine scheduling latency. High goroutine scheduling latency is often the cause of node instability.

The scheduling latency is correlated with periodic high rates of Get requests that we believe are the result of all nodes in the cluster calling IsAlive on the sql liveness session of every other node with a small window of time (the SQL liveness heartbeat interval).

DistSQL now uses sql instances based planning by default and the code calls GetAllInstances

For example:

https://github.com/cockroachdb/cockroach/blob/838bcb171017937a827d5aab1eba9fc4f31732c6/pkg/sql/distsql_physical_planner.go#L1885

This method will check the liveness record of every node:

https://github.com/cockroachdb/cockroach/blob/838bcb171017937a827d5aab1eba9fc4f31732c6/pkg/sql/sqlinstance/instancestorage/instancereader.go#L254-L287

We can see that turning off SQL-instance based planning stops the bursty traffic, seemingly confirming this theory

Image

Reduced the Get traffic and reduced the p99 goroutine scheduling latency.

*Open questions*

- A suspicion is the SQL AUTO STATS job may be one of the main drivers of these requests. We noted that when a long-running SQL AUTO STATS jobs was running (and thus blocking the steady stream of short-lived auto stats jobs typically starting and finishing regularly in the cluster), we see the same drop in requests and scheduling latency. Our suspicion is that the DistSQL plans used by auto stats are more likely to hit these code paths in DistSQL. It would be good to be able to explain this exactly.

- Why does this request rate have such a large impact? We note that the transactions used by IsAlive bypass AC.

Jira issue: CRDB-51213

Epic CRDB-52349

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.