multi-node citus coordinator can't connect to worker nodes on macos
- Dominant language
- C
- Stars
- 12.8k
- Forks
- 794
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 31
Description
Bug description:
We are trying to create a Citus cluster with 1 coordinator and 2 workers on macOS. The coordinator listens to port 5432 and 2 workers listen to 5433 and 5434.
After installing citus and executing all steps according to'Multi-Node Citus' part in official docs, citus_add_node command failed with error message `ERROR: connection establishment for node localhost:5433 failed \ HINT: Check both the local and remote server logs for the connection establishment errors.`
Citus version is Citus 12.1devel on arm-apple-darwin21.6.0, compiled by Apple clang version 14.0.0 (clang-1400.0.29.202), 64-bit gitref: main(sha: f03291a8c). With postgresql-15.4 installed by homebrew.
Causes:
We try to add EmitErrorReport(); to pg_catch() block in function CitusModifyWaitEvent() in connection_management.c and recompiled citus. It shows that error happens in WaitEventAdjustKqueue() in pg with error message `kevent() failed: no such file or directory`.
We figure out that the real cause is that PQconnectPoll may return PGRES_POLLING_READING or PGRES_POLLING_WRITING when connecting. In this case, return value of MultiConnectionStatePoll is true and connectionState->phase == MULTI_CONNECTION_PHASE_CONNECTING, so CitusModifyWaitEvent will be called. ModifyWaitEvent will call WaitEventAdjustKqueue, which will call kevent on invalid kqueue_fd of not really established connection, and finally causing errors.
Some suggested fix:
Replacing asynchronous PQconnectStartParams() in connection_management.c to synchronous PQconnectdbParams() will definitely solve this bug, but may harm performance when having a lot of connections.
Maybe citus could follow the way that PQconnectPoll is used inside the source code of PostgreSQL, for example, like in libpqwalreceiver.c, call PQconnectPoll in a do-while loop until it returns PGRES_POLLING_OK or PGRES_POLLING_FAILED , or in for(;;) loop just like in fe-connect.c
Or just simply avoid call CitusModifyWaitEvent() before the connection is established when using kqueue.
By the way, we suggest add EmitErrorReport(); to pg_catch block before FlushErrorState(); so that we can see the real error message from postgresql.
Contributor guide
Assessment
This issue has not been assessed yet.