Broker doesn't get killed when the shutdown exceeds brokerShutdownTimeoutMs
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
**Describe the bug**
Broker should get killed after brokerShutdownTimeoutMs, which is 60 seconds by default.
```
# Time to wait for broker graceful shutdown. After this time elapses, the process will be killed
brokerShutdownTimeoutMs=60000
```
I have noticed that sometimes broker pods don't get stopped and pods will have to be manually killed.
**To Reproduce**
I haven't found a way to reproduce the problem.
**Expected behavior**
Broker should get killed after brokerShutdownTimeoutMs
**Additional context**
It looks like the brokerShutdownTimeoutMs feature stopped working as expected when [MessagingServiceShutdownHook](https://github.com/apache/pulsar/blob/9516e5db1a250d39101d0c7acda6d1fd8366c8d3/pulsar-broker/src/main/java/org/apache/pulsar/broker/MessagingServiceShutdownHook.java) got removed in https://github.com/apache/pulsar/pull/12146 .
It would be useful to print out the thread dump to System.out before killing the process with `Runtime.getRuntime().halt(1)`. This could help investigate why the shutdown is stuck.
```java
System.out.println(ThreadDumpUtil.buildThreadDiagnosticString());
```
Contributor guide
Research direction
Start by reading pulsar-broker/src/main/java/org/apache/pulsar/broker/MessagingServiceShutdownHook.java history and the current shutdown entry point, then inspect the brokerShutdownTimeoutMs handling. Check how ThreadDumpUtil.buildThreadDiagnosticString() could support diagnosing a stuck shutdown. Done means the broker is killed after the configured timeout and the relevant shutdown behavior is covered by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100