Altinity / Altinity/clickhouse-operator
Provide the ability to add custom health checks that precede and follow ClickHouse reconfiguration/upgrade
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 2.6k
- Forks
- 574
- Avg merge
- 8d 6h
- Merged PRs (30d)
- 6
Description
We had a recent discussion with a user who has the following use case. They have an operator that watches for replication lag on ClickHouse and only schedules upgrade when it's below a certain level. They were wondering if the operator could help them implement logic that amount to "guard conditions" on operator upgrades. It could work as follows:
- Permit a ClickHouse replica upgrade operation to start when replication lag is below a "reasonable" level.
- Confirm that the replication lag is back to a reasonable level before moving to the next replica.
In this case the guard condition is replica lag but it's likely that there would be others. They and other users would presumably want to customize the conditions.
It seems as if this could be implemented in a general way by adding a custom health check section to CHI instances that contain one or more SQL queries to run before and after upgrade. If the check(s) pass, the upgrade could proceed. If they fail the operator would wait.
There are some questions with this kind of guard that we need to contemplate.
- How would other services learn the status of the cluster health checks?
- What happens if replica lag increases suddenly during an upgrade? It could block an incomplete operation indefinitely.
- Does it just apply to upgrade or to any reconfiguration that causes a pod restart?
- Should we report these health check results in our exported metrics? It would be important to see these visually in, say, Grafana when kicking off an upgrade or diagnosing problems.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are named. Start by reviewing the four design questions in the issue and the operator's existing reconfiguration and upgrade behavior. Done means an agreed design for custom pre- and post-operation checks, failure handling, status visibility, and metric reporting.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- databases, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100