citusdata / citusdata/citus

multi-node citus coordinator can't connect to worker nodes on macos

Open
#7,168 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
12.8k
Forks
794
Avg merge
2d 14h
Merged PRs (30d)
31

Description

Bug description:
We are trying to create a Citus cluster with 1 coordinator and 2 workers on macOS. The coordinator listens to port 5432 and 2 workers listen to 5433 and 5434.
After installing citus and executing all steps according to'Multi-Node Citus' part in official docs, citus_add_node command failed with error message `ERROR: connection establishment for node localhost:5433 failed \ HINT: Check both the local and remote server logs for the connection establishment errors.`

Citus version is Citus 12.1devel on arm-apple-darwin21.6.0, compiled by Apple clang version 14.0.0 (clang-1400.0.29.202), 64-bit gitref: main(sha: f03291a8c). With postgresql-15.4 installed by homebrew.

Causes:
We try to add EmitErrorReport(); to pg_catch() block in function CitusModifyWaitEvent() in connection_management.c and recompiled citus. It shows that error happens in WaitEventAdjustKqueue() in pg with error message `kevent() failed: no such file or directory`.

We figure out that the real cause is that PQconnectPoll may return PGRES_POLLING_READING or PGRES_POLLING_WRITING when connecting. In this case, return value of MultiConnectionStatePoll is true and connectionState->phase == MULTI_CONNECTION_PHASE_CONNECTING, so CitusModifyWaitEvent will be called. ModifyWaitEvent will call WaitEventAdjustKqueue, which will call kevent on invalid kqueue_fd of not really established connection, and finally causing errors.

Some suggested fix:
Replacing asynchronous PQconnectStartParams() in connection_management.c to synchronous PQconnectdbParams() will definitely solve this bug, but may harm performance when having a lot of connections.

Maybe citus could follow the way that PQconnectPoll is used inside the source code of PostgreSQL, for example, like in libpqwalreceiver.c, call PQconnectPoll in a do-while loop until it returns PGRES_POLLING_OK or PGRES_POLLING_FAILED , or in for(;;) loop just like in fe-connect.c

Or just simply avoid call CitusModifyWaitEvent() before the connection is established when using kqueue.

By the way, we suggest add EmitErrorReport(); to pg_catch block before FlushErrorState(); so that we can see the real error message from postgresql.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.