Altinity / Altinity/clickhouse-operator
Operator normalizes single-element grant arrays to strings, causing Helm upgrade conflicts
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 2.6k
- Forks
- 574
- Avg merge
- 8d 6h
- Merged PRs (30d)
- 6
Description
The ClickHouse Operator appears to normalize single-element arrays in the spec.configuration.users.<username>/grants/query field to string values during reconciliation. This causes Helm upgrade conflicts when later adding more grants to the array.
Environment
- ClickHouse Operator Version: 0.25.5
- Kubernetes Version: (your version)
- Helm Version: (your version)
- Deployment Method: Helm chart
Steps to Reproduce
- Initial deployment with a single grant:
Deploy a CHI resource via Helm with one grant:
spec:
configuration:
users:
metabase/grants/query:
- "GRANT SELECT ON logs.*"
- Check the stored resource:
kubectl get chi clickhouse -n observability -o yaml | grep -A 5 "metabase/grants"
Expected: Array format
metabase/grants/query:
- "GRANT SELECT ON logs.*"
Actual: String format (after operator reconciliation)
metabase/grants/query: "GRANT SELECT ON logs.*"
- Attempt to add more grants:
Update the Helm values to include multiple grants:
spec:
configuration:
users:
metabase/grants/query:
- "GRANT SELECT ON logs.*"
- "GRANT SELECT ON system.processes"
- "GRANT KILL QUERY ON *.*"
- Run Helm upgrade:
helm upgrade clickhouse ./chart -n observability
Result: Helm reports a conflict error:
Error: UPGRADE FAILED: conflict occurred while applying object observability/clickhouse
clickhouse.altinity.com/v1, Kind=ClickHouseInstallation:
Apply failed with 1 conflict: conflict with "helm" using clickhouse.altinity.com/v1:
.spec.configuration.users.metabase/grants/query
Root Cause Analysis
The issue occurs because:
- The CRD uses
x-kubernetes-preserve-unknown-fields: truefor theusersfield - When a single-element array is deployed, the operator normalizes it to a string during reconciliation
- When Helm tries to update from string to array, it detects a type incompatibility and reports a conflict
- Helm's three-way strategic merge cannot automatically resolve type changes (string → array)
Impact
- Helm deployments fail when adding additional grants to users that initially had only one grant
- Requires manual intervention (kubectl patch) to resolve
- Inconsistent behavior: 2+ element arrays are preserved, but single-element arrays are converted
- Breaks GitOps workflows where configuration is managed declaratively through Helm
Workaround
Option 1: Always use 2+ grants (add a harmless grant):
users:
- name: metabase
grants:
- "GRANT SELECT ON logs.*"
- "GRANT SHOW ON *.*" # Dummy grant to prevent normalization
Option 2: Use kubectl patch to update grants:
kubectl patch chi clickhouse -n observability --type=json -p='[...]'
Expected Behavior
The operator should preserve the array format even for single-element arrays, to maintain consistency and avoid Helm conflicts.
Suggested Solutions
- Disable single-element array normalization for grant fields
- Add a configuration option to control normalization behavior
- Document this behavior prominently in the operator documentation
- Preserve original data types from the API server without modification
Additional Context
This behavior is particularly problematic for teams using:
- Helm for infrastructure-as-code
- GitOps workflows (ArgoCD, Flux)
- Gradual permission expansion (starting with minimal grants)
The normalization might be intended for optimization, but it creates operational challenges in production environments where declarative configuration management is critical.
Related Code
The issue likely originates in the operator's reconciliation logic where it processes the users configuration and generates the ConfigMap for ClickHouse.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the reconciliation logic that processes the users configuration and generates the ClickHouse ConfigMap; no specific file or test is identified in the issue. Reproduce the Helm deployment and upgrade sequence with one grant becoming multiple grants, then verify that reconciliation preserves the array format and the upgrade no longer reports a conflict.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, helm, kubernetes
- Domain
- databases, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100