Citus does not set a read/write timeout when communicating with remote nodes
- Dominant language
- C
- Stars
- 12.8k
- Forks
- 794
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 31
Description
Working off of [this gist](https://gist.github.com/saicitus/24e53577a11662ebf21a4163a371da79).
A prospective user ran into a problem while testing Citus where, when they (effectively) drop packets between the master and a worker, the master stalls while waiting for that worker to respond. If the connection is explicitly dropped Citus will failover to the next worker but it will otherwise sit idly until `libpq` timeout.
This is all fine, except `libpq` defaults to using the system's TCP keepalive settings for timeout which can be rather large indeed.
[EDIT: added the below after thinking about this more]
We should [set our own timeout](http://choly.ca/post/libpq-pqexec-timeout/) when calling PQexec. Even if we used reasonable settings for keep-alive, that would not save us from the master locking up in case of worker deadlocks.
Contributor guide
Assessment
This issue has not been assessed yet.