cockroachdb / cockroachdb/cockroach
sql,kv: sqlinstances based planning causes high goroutine scheduling latency because of bursty system.sqlliveness traffic
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
In a large scale test, we noted that the leaseholder of the range holding the `system.sqlliveness` table was consistently observing periods of high goroutine scheduling latency. High goroutine scheduling latency is often the cause of node instability.
The scheduling latency is correlated with periodic high rates of Get requests that we believe are the result of all nodes in the cluster calling IsAlive on the sql liveness session of every other node with a small window of time (the SQL liveness heartbeat interval).
DistSQL now uses sql instances based planning by default and the code calls GetAllInstances
For example:
https://github.com/cockroachdb/cockroach/blob/838bcb171017937a827d5aab1eba9fc4f31732c6/pkg/sql/distsql_physical_planner.go#L1885
This method will check the liveness record of every node:
https://github.com/cockroachdb/cockroach/blob/838bcb171017937a827d5aab1eba9fc4f31732c6/pkg/sql/sqlinstance/instancestorage/instancereader.go#L254-L287
We can see that turning off SQL-instance based planning stops the bursty traffic, seemingly confirming this theory
Reduced the Get traffic and reduced the p99 goroutine scheduling latency.
*Open questions*
- A suspicion is the SQL AUTO STATS job may be one of the main drivers of these requests. We noted that when a long-running SQL AUTO STATS jobs was running (and thus blocking the steady stream of short-lived auto stats jobs typically starting and finishing regularly in the cluster), we see the same drop in requests and scheduling latency. Our suspicion is that the DistSQL plans used by auto stats are more likely to hit these code paths in DistSQL. It would be good to be able to explain this exactly.
- Why does this request rate have such a large impact? We note that the transactions used by IsAlive bypass AC.
Jira issue: CRDB-51213
Epic CRDB-52349
Contributor guide
Assessment
This issue has not been assessed yet.