redpanda-data / redpanda-data/redpanda

Support >1 node in maintenance mode in multi-rack/AZ clusters

Open
#15,084 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

kind/enhance
Dominant language
C++
Stars
12.6k
Forks
792
PR merge metrics
No merged PRs in 30d

Description

Discussion: https://redpandacommunity.slack.com/archives/C01B3QBLUTT/p1700222269091209

Who is this for and what problem do they have today?

Maintenance mode is capped at 1 node at a time to preserve data availability and safety. Additionally, the command is susceptible to race conditions

Orchestrating maintenance/restarts/upgrades across multiple nodes in parallel is desirable for large clusters to speed updates & upgrades. This should be possible if the command is topology-aware (ensures rack awareness is enabled and the full set of nodes in maintenance mode at any time never brings available partition replicas below a majority for any partitions on those nodes).

That said, capacity constraints come into play for production clusters, and one may not want the entire fault domain to go down with every config change/restart. Often distributed systems handle this problem with a configurable batch size.

What are the success criteria?

As an admin, on a cluster with multiple fault domains, I can safely put all nodes within a given fault domain into maintenance mode at the same time (with guardrails to prevent any loss of data availability, rejecting the operation as required)
As an admin, I can 'throttle' the amount of nodes that go offline at once during and update/upgrade, by configuring a batch size

We should make no change to the defaults here in redpanda (still 1 node for safety), but make it a tunable cluster configuration property (e.g. rolling_restart_batch_size)

JIRA Link: CORE-1589

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names no source files or tests. Begin by tracing the maintenance-mode command and the cluster configuration entry points, then identify how rack or fault-domain awareness and replica availability are checked. Done means safely supporting a configurable batch size while retaining a default of one and rejecting operations that would reduce availability below a majority.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
distributed-systems, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.