opensearch-project / opensearch-project/OpenSearch

Support for decommissioning and recommissioning a zone

Open
#3,402 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

discuss enhancement
Dominant language
Java
Stars
13.7k
Forks
3k
Avg merge
2d 23h
Merged PRs (30d)
108

Description

Is your feature request related to a problem? Please describe.
There are use-case where decommissioning a zone/rack might be beneficial

  1. In a multi-zone deployment setup it might be good to support zonal deployment rather than a rolling restart per node, which might be too slow for a big cluster and might take sufficiently longer. In such a case zone serves a unit of deployment
  2. Andon cord during zonal outages is another handy mechanism which enables graceful traffic shutdown in the impacted zone/rack espl when certain nodes are still operating in a degraded manner

Background
With #2859 we intend to weigh away shard search traffic, however since OpenSearch follows a synchronous replication model, it is possible to have replication request stuck due to any impairment in the write path. The current health check mechanisms to detect and remediate a bad node is only a best effort strategy and doesn't cover deeper health checks across all network paths. For predictability, we propose pulling an andon cord to cut-off inter-zone replication traffic, which can be achieved by decommissioning the node in the impacted zone.

Implications
As a result of decommissioning a zone all shards that were taking in write traffic might fail to ensure data consistency semantics are honoured and stale shards are marked unavailable. To make sure no in-flight requests fail we need to weigh away shard search traffic as a part of #2859. In some setups where there are no dedicated coordinator setups we need to ensure no HTTP traffic is being sent and all traffic is drained before a decommission API is triggered

Describe the solution you'd like
A graceful mechanism to

  1. Decommission a zone/rack
  2. Recommission a zone/rack
POST /_cluster/decommission 
{
 "awareness_attribute" : {"zone" : "A-0"}
}
DELETE /_cluster/decommission 
{
 "awareness_attribute" : {"zone" : "A-0"}
}

Constraints

  1. Attribute value should be a one of the values in union(forces_zone, discovered_zone)
  2. There should be only one active zone under decommission or recommission
  3. The shard request weights on the decommissioned zone from #2859 should be set to zero

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the context from #2859 and the proposed POST and DELETE /_cluster/decommission API examples. Define how a zone or rack is gracefully decommissioned and recommissioned while enforcing the listed attribute, single-active-zone, and zero-weight constraints; done means the specified API behavior and traffic-draining semantics are covered.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.