Document recommended Kubernetes maintenance workflows
- Dominant language
- Python
- Stars
- 27
- Forks
- 9
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 11
Description
Customers operating CrateDB through crate-operator need guidance for safely upgrading or replacing Kubernetes worker nodes.
In a recent support case, a customer reported unavailability of data (missing primary shards) during an upgrade of Kubernetes nodes. Even though such events aren't triggered by the operator, customers still seek advice from us as it concerns deployments created by the operator.
The operator currently defines a disruption budget with `max_unavailable=1`, meaning only one pod can be unavailable at a time:
https://github.com/crate/crate-operator/blob/2d05138d2f41dda617ded37d8bb9385b5afafe55/crate/operator/create.py#L1122-L1160
This ensures Kubernetes waits until an unavailable pod has become ready again, but a ready pod doesn't equal a ready CrateDB (all [primary] shards recovered). When strictly following this metric, data availability in CrateDB is not guaranteed.
For such externally initiated Kubernetes maintenance, there is no documentation explaining how to run in in a safe way, so that:
1. Kubernetes hosts are upgraded/replaced without CrateDB entering a red state.
2. No unnecessary shard recovery/rebalancing happens while Kubernetes nodes are restarted.
3. Users know how to determine when it is safe to drain the next Kubernetes host.
A documented procedure similar to the [Rolling Upgrade Guide](https://cratedb.com/docs/guide/admin/upgrade/rolling.html), addressing how to perform a Kubernetes upgrade safely, addressing above points.
---
### How Elasticsearch approaches it
Elasticsearch [dynamically updates the PDB](https://www.elastic.co/docs/deploy-manage/deploy/cloud-on-k8s/pod-disruption-budget#default-pdb-rules):
> In Elastic Cloud on Kubernetes clusters that do not have an Enterprise license, one Elasticsearch Pod can be taken down at a time, as long as the cluster has a health status of green. Single-node clusters are not considered highly available and can always be disrupted.
If a node is taken down, the PDB is updated to disallow the removal of additional pods. It changes the PDB back again once the cluster is in a green state.
Contributor guide
Research direction
Start with the linked Rolling Upgrade Guide and the PDB definition in operator/create.py around lines 1122-1160. Document a Kubernetes worker-node upgrade procedure covering safe draining, avoiding unnecessary shard recovery or rebalancing, and checking when the next host can be drained without CrateDB entering a red state.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes
- Domain
- databases, devops, documentation
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 74/100