apache / apache/couchdb

Handle SIGTERM on CouchDB Cluster

Open
#4,173 0 comments 0 reactions 0 assignees View on GitHub
enhancement needs-triage
Dominant language
Erlang
Stars
7k
Forks
1.1k
Avg merge
1d 16h
Merged PRs (30d)
9

Description

I am stress testing for a high availability environment of a Couch cluster on a Kubernetes cluster. The issue I am currently seeing is that when a Couch cluster node (kubernetes pod) is removed, in the time it is being removed and recreated, a series of 5xx errors are thrown for requests that are alive in that interval. Going a little deeper, I am trying to understand if Couch handles some mechanism for detecting the SIGTERM signal from the main process, in order to divert the live connections in the node that is going to be recreated to other nodes in the cluster and avoid receiving 5xx errors.

Is there a mechanism to handle SIGTERM signals in a Couch cluster? If not, it would be an interesting feature that would allow us to have more control to avoid losing connections at the time of node loss.

Contributor guide

Open the contributing guide

Research direction

Start by reading the CouchDB cluster and Kubernetes shutdown behavior relevant to SIGTERM, then reproduce removal and recreation of a node while requests remain active. Determine whether live connections can be diverted to other nodes and define done as avoiding the reported 5xx errors during node loss.

Written by the indexing model from the issue text.

Assessment

Tech stack
erlang, kubernetes
Domain
cloud, databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.