Clickhouse Operator Plugin Architecture

Introduction

1. As the ClickHouse Operator matures, its codebase continues to grow—introducing complexity and slowing the delivery of new features like backup/restore, shard rebalancing and other critical capabilities.
2. To address this, we propose introducing a plugin-based architecture—similar to CloudNativePG’s CNPG-I—enabling independent extensions to the operator through a well-defined interface.
3. This approach allows the ClickHouse Operator to remain lightweight and focused, while empowering the community to innovate and extend functionality via external, modular plugins.

Goals

1. Introduce a modular and extensible approach to integrate plugins with the Altinity Clickhouse operator, enabling greater flexibility and ease of feature development in the Clickhouse Operator ecosystem.
2. Define a gRPC-based protocol that defines a standardized interface, named ClickhouseOperatorPlugin-Interface(COP-I), between the Altinity operator and external plugins.
3. Empower developers and organizations to extend the operator's functionality without the need to fork its codebase.

Key Benefits

1. Reduced Core Complexity: Offload auxiliary responsibilities—like backup orchestration or rebalancing logic—to external plugins, keeping the core operator clean and maintainable.
2. Accelerated Feature Development: Enable faster delivery of features by allowing parallel plugin development without needing changes to the main operator.
3. Standardized Plugin Interface: Define a gRPC API contract for plugins, enabling safe and consistent integration.
4. Ecosystem Growth: Encourage external contributions by making it easier to build and share plugins, avoiding forks and duplication.
5. Customization Without Forking: Support advanced user needs (e.g., custom backup tools) via pluggable extensions.

Use Cases for Plugins

1. A modular ClickHouse Operator architecture would support plugins for: Backup and Restore (e.g., to S3)
2. Shard Rebalancing and Resharding Strategies
3. Cluster Lifecycle Hooks (pre/post scale, failover events)
4. Integration with External Systems

Plugin Architecture

1. The operator provides a gRPC service interface for all auxiliary features (e.g., backup, re-sharding).
2. Each plugin implements the gRPC service for a specific feature.
3. The plugin will be deployed separately from the operator
4. The operator will discover the plugin using labels and annotations of the plugin’s K8s service

Block Diagram

The above diagram illustrates two plugins:

1. Backup Plugin – Executes SQL commands on the ClickHouse cluster to back up data to S3.
2. Resharding/Rebalance Plugin – Redistributes data across the ClickHouse cluster for load balancing.

Start by reviewing the operator's current architecture and the proposed Kubernetes Service labels and annotations for plugin discovery. Define the COP-I gRPC contract, plugin deployment and discovery boundaries, and how backup or resharding plugins would integrate. Done means the architecture and interface are specified well enough to guide independent plugin implementations.