Graylog2 / Graylog2/graylog2-server

Node stuck in graceful shutdown keeps the leader lock, blocking failover of leader-only work

Open
#27,382 2 comments 0 reactions 0 assignees View on GitHub
bug triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

## Problem

A node that hangs during graceful shutdown keeps the leader lock forever. No other node can take over, so leader-only work has no way to fail over for as long as the process stays alive.

## Observed case

Two node 7.2.0-beta.2 test cluster. The leader received SIGTERM and was still shutting down 50 minutes later. The heartbeat stays fresh because `NodePingThread.stopOnGracefulShutdown()` returns `false` by design, so the node keeps advertising itself as leader while it is dying.

What made the hang unbounded: the search cluster was in flood stage with indices set to `read_only_allow_delete`. Writes were rejected, so the output buffer backed up, the process buffer filled, the input buffer filled, and the inputs could not finish stopping. The shutdown never got past `inputSetupService.awaitTerminated()`.

## Proposed fix

Release the leader lock at the `serverStatus.shutdown()` transition, before the REST API and the inputs are stopped. At that point the node has already set `lb_status` to `DEAD` and has committed to dying. Everything after is best effort draining.

The handover path already exists and does not need to be built:

1. Releasing the lock posts `LeaderChangedEvent`.
2. `PeriodicalsService.leaderChanged()` sees `isLeader() == false` and calls `stopPeriodicals(runningLeaderNodePeriodicals)` on the dying node.
3. The next node to acquire the lock starts them.

## Risk analysis

**Risk of leaving it: medium to high. Risk of fixing it: low.**

It is rare, but it happens during outages and rolling restarts rather than at random, so it shows up when the cluster is already in trouble. Once it happens nothing recovers on its own and someone has to kill the process by hand.

The fix reuses a handover that already happens today, so it is a small change with one hazard that is easy to catch in review.

## Companion change

Put a deadline on the three unbounded `awaitTerminated()` calls in `GracefulShutdown`. A node that has told the load balancer it is dead should have a hard limit on how long it is allowed to take to actually die. This bounds every consequence of a stuck shutdown rather than just the leadership one, and the two changes are independent.

## How to reproduce

1. Start a two node cluster.
2. Fill the search cluster disk past the flood stage watermark so indices go `read_only_allow_delete`.
3. Send enough traffic to fill the input buffer on the leader.
4. Send SIGTERM to the leader.
5. The leader hangs in shutdown, keeps `is_leader: true`, and the other node never takes over.

Contributor guide

Open the contributing guide

Research direction

Start by tracing GracefulShutdown and the serverStatus.shutdown() transition, then read NodePingThread.stopOnGracefulShutdown() and PeriodicalsService.leaderChanged() to understand the existing handover. Verify the lock is released before REST and input shutdown, and bound the three awaitTerminated() calls with deadlines; add or update focused tests for leadership handover and stuck shutdown behavior if the surrounding test structure supports them.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.