citusdata / citusdata/citus

Connection management improvements for multi-database setup

Open
#7,244 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
12.8k
Forks
794
Avg merge
2d 14h
Merged PRs (30d)
31

Description

This issue is more of an invitation to a technical discussion than a solid proposal.

Currently, the main issue with having multiple `DATABASE`s in a cluster is connection management which is not adapted to this. The issue can be split into two parts:

1. The most problematic areas are the transaction recovery and distributed deadlock detection mechanisms that do not respect `citus.max_shared_pool_size` and open at least `n * d` connections per worker, where n -- number of nodes in cluster, d -- number of `DATABASE`s
2. Even if they respect the `citus.max_shared_pool_size`, this limit is applied per `DATABASE`, meaning that it will still require `n * d` connections.

In order to overcome this, I propose to:

1. Make `citus.max_shared_pool_size` cluster-wide. Per-database setting then may be used to set a connection quota for a database within the global `citus.max_shared_pool_size`
2. Make the transaction recovery and distributed deadlock detection respect the improved `citus.max_shared_pool_size`.

Since those changes make sense mostly for the multi-database setup, they may be enabled by a single GUC, like `citus.multi_database`, or have a separate GUC for the behaviour of `citus.max_shared_pool_size`, transaction recovery, and distributed deadlock detection.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.