citusdata / citusdata/citus

Distributed deadlock detection should skip checks when there are no blocked processes

Open
#1,925 0 comments 2 reactions 0 assignees View on GitHub
research usability
Dominant language
C
Stars
12.8k
Forks
794
Avg merge
2d 14h
Merged PRs (30d)
31

Description

With 7.0 (#1394), we've introduced distributed deadlock detection. The current implementation keeps running the deadlock detection for every `deadlock_timeout * citus.distributed_deadlock_detection_factor` msecs. The default is `1 secs * 2 = 2 seconds`.

We should be able to skip distributed deadlock detection if there are no blocked processes on the coordinator (or MX-worker). The reason is that, if there are no blocked processes, it is not possible to end up with distributed deadlocks.

Note that, **blocked** in this scope doesn't necessarily mean processes blocked on a lock. The coordinator processes might be waiting for a response on the worker nodes (which is actually blocked on a lock).

There seem to be two benefits of this:
* Avoid CPU and connection overhead (which are not high, but still there is some)
* Get rid of the very frequent ` SELECT * FROM dump_local_wait_edges()` logs on the
workers when `log_min_duration_statement` is very small (i.e., `-1` to log all commands executed
on the node)

The complete solution to this problem seems relatively complex given that we need to go over all the active coordinator processes and understand whether the corresponding worker processes are waiting for locks on the workers or not.

The non-complete solution, that'd alleviate the second item in the above list for certain cases* could be to skip distributed deadlock detection when there are no active distributed transactions.

* empty cluster or router `SELECT`s along with single shard modifications that are not in transaction blocks

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.