crate / crate/crate-operator

Document recommended Kubernetes maintenance workflows

Open
#903 0 comments 0 reactions 0 assignees View on GitHub
documentation triage
Dominant language
Python
Stars
27
Forks
9
Avg merge
3d 1h
Merged PRs (30d)
11

Description

Customers operating CrateDB through crate-operator need guidance for safely upgrading or replacing Kubernetes worker nodes.
In a recent support case, a customer reported unavailability of data (missing primary shards) during an upgrade of Kubernetes nodes. Even though such events aren't triggered by the operator, customers still seek advice from us as it concerns deployments created by the operator.

The operator currently defines a disruption budget with `max_unavailable=1`, meaning only one pod can be unavailable at a time:

https://github.com/crate/crate-operator/blob/2d05138d2f41dda617ded37d8bb9385b5afafe55/crate/operator/create.py#L1122-L1160

This ensures Kubernetes waits until an unavailable pod has become ready again, but a ready pod doesn't equal a ready CrateDB (all [primary] shards recovered). When strictly following this metric, data availability in CrateDB is not guaranteed.

For such externally initiated Kubernetes maintenance, there is no documentation explaining how to run in in a safe way, so that:
1. Kubernetes hosts are upgraded/replaced without CrateDB entering a red state.
2. No unnecessary shard recovery/rebalancing happens while Kubernetes nodes are restarted.
3. Users know how to determine when it is safe to drain the next Kubernetes host.

A documented procedure similar to the [Rolling Upgrade Guide](https://cratedb.com/docs/guide/admin/upgrade/rolling.html), addressing how to perform a Kubernetes upgrade safely, addressing above points.

---

### How Elasticsearch approaches it

Elasticsearch [dynamically updates the PDB](https://www.elastic.co/docs/deploy-manage/deploy/cloud-on-k8s/pod-disruption-budget#default-pdb-rules):

> In Elastic Cloud on Kubernetes clusters that do not have an Enterprise license, one Elasticsearch Pod can be taken down at a time, as long as the cluster has a health status of green. Single-node clusters are not considered highly available and can always be disrupted.

If a node is taken down, the PDB is updated to disallow the removal of additional pods. It changes the PDB back again once the cluster is in a green state.

Contributor guide

Open the contributing guide

Research direction

Start with the linked Rolling Upgrade Guide and the PDB definition in operator/create.py around lines 1122-1160. Document a Kubernetes worker-node upgrade procedure covering safe draining, avoiding unnecessary shard recovery or rebalancing, and checking when the next host can be drained without CrateDB entering a red state.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes
Domain
databases, devops, documentation
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.