citusdata / citusdata/citus

Query on Best Practices for Handling High Bandwidth on the Coordinator in Large Citus Clusters

Open
#7,654 0 comments 0 reactions 0 assignees View on GitHub
Support
Dominant language
C
Stars
12.8k
Forks
794
Avg merge
2d 14h
Merged PRs (30d)
31

Description

Hello Citus Community,

I am currently designing a Citus cluster expected to manage over 100TB of data and handle millions of queries. Given the high traffic anticipated, I am concerned about the potential for the coordinator to become a bottleneck, despite the scalability improvements in Citus 11.

According to the [Citus documentation](https://docs.citusdata.com/en/v9.3/arch/mx.html), the coordinator is primarily responsible for storing metadata and final aggregations, and while it's possible to add another coordinator, it doesn't mention handling multiple primary coordinators.

With this setup:
- Is there a recommended approach or best practices for managing high bandwidth impacts on the coordinator?
- Could you provide insights or examples of how other large-scale deployments have optimized coordinator performance under similar conditions?

Any advice or guidance would be greatly appreciated as we aim to optimize our architecture for high performance and reliability.

Thank you!

Contributor guide

Open the contributing guide

Research direction

Start with the linked Citus documentation on the coordinator and the issue's references to Citus 11 and multiple coordinators. Determine whether the project has documented guidance or deployment examples for coordinator bandwidth at large scale; done would require a concrete, maintainable answer or documentation update, but no target file or test is identified.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, sql
Domain
databases, distributed-systems
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.