citusdata / citusdata/citus

Citus does not set a read/write timeout when communicating with remote nodes

Open
#1,135 9 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
C
Stars
12.8k
Forks
794
Avg merge
2d 14h
Merged PRs (30d)
31

Description

Working off of [this gist](https://gist.github.com/saicitus/24e53577a11662ebf21a4163a371da79).

A prospective user ran into a problem while testing Citus where, when they (effectively) drop packets between the master and a worker, the master stalls while waiting for that worker to respond. If the connection is explicitly dropped Citus will failover to the next worker but it will otherwise sit idly until `libpq` timeout.

This is all fine, except `libpq` defaults to using the system's TCP keepalive settings for timeout which can be rather large indeed.

[EDIT: added the below after thinking about this more]

We should [set our own timeout](http://choly.ca/post/libpq-pqexec-timeout/) when calling PQexec. Even if we used reasonable settings for keep-alive, that would not save us from the master locking up in case of worker deadlocks.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.