Altinity / Altinity/clickhouse-operator
Large Cluster Configmap Limits
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 2.6k
- Forks
- 574
- Avg merge
- 8d 6h
- Merged PRs (30d)
- 6
Description
Hi All,
I recently hit an issue where I was trying to provision a large Clickhouse cluster (1000 shards) and hit a limit on the amount of nodes that could exist in the relevant configmap. After 807 nodes are provisioned the configmap is larger than 1MB in size which is a hard limit in K8S/etcd. I am wondering if there is a workaround or ideas on how to partition the config maps appropriately to work around this issue.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by examining how the operator stores ClickHouse cluster node data in Kubernetes ConfigMaps. Reproduce the reported limit with a large cluster and investigate partitioning the ConfigMaps so they remain below Kubernetes' 1 MB limit. Done means a 1000-shard cluster can be provisioned without exceeding the ConfigMap or etcd size limit.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, go, kubernetes
- Domain
- databases, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100