apache / apache/pulsar

Broker doesn't get killed when the shutdown exceeds brokerShutdownTimeoutMs

Open
#15,650 7 comments 0 reactions 0 assignees View on GitHub
area/broker Stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

**Describe the bug**

Broker should get killed after brokerShutdownTimeoutMs, which is 60 seconds by default.

```
# Time to wait for broker graceful shutdown. After this time elapses, the process will be killed
brokerShutdownTimeoutMs=60000
```

I have noticed that sometimes broker pods don't get stopped and pods will have to be manually killed.

**To Reproduce**

I haven't found a way to reproduce the problem.

**Expected behavior**

Broker should get killed after brokerShutdownTimeoutMs

**Additional context**

It looks like the brokerShutdownTimeoutMs feature stopped working as expected when [MessagingServiceShutdownHook](https://github.com/apache/pulsar/blob/9516e5db1a250d39101d0c7acda6d1fd8366c8d3/pulsar-broker/src/main/java/org/apache/pulsar/broker/MessagingServiceShutdownHook.java) got removed in https://github.com/apache/pulsar/pull/12146 .

It would be useful to print out the thread dump to System.out before killing the process with `Runtime.getRuntime().halt(1)`. This could help investigate why the shutdown is stuck.
```java
System.out.println(ThreadDumpUtil.buildThreadDiagnosticString());
```

Contributor guide

Open the contributing guide

Research direction

Start by reading pulsar-broker/src/main/java/org/apache/pulsar/broker/MessagingServiceShutdownHook.java history and the current shutdown entry point, then inspect the brokerShutdownTimeoutMs handling. Check how ThreadDumpUtil.buildThreadDiagnosticString() could support diagnosing a stuck shutdown. Done means the broker is killed after the configured timeout and the relevant shutdown behavior is covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.