apache / apache/pulsar

[2.8.0] After being killed with SIGKILL, pulsar standalone will not start unless its data directory is removed.

Open
#11,842 4 comments 2 reactions 0 assignees View on GitHub
lifecycle/stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

**Describe the bug**
If I kill Pulsar standalone with SIGKILL/`kill -9` after it has been started, the broker will not successfully start anytime thereafter unless its data directory is removed and it is reset to initial state.

If this isn't easily fixable, please provide a workaround that I can use on Pulsars that get into this state to bring them back without deleting all their data (something I can do in ZK or to the BK data files short of deleting everything that makes startup succeed).

**To Reproduce**
1. Start pulsar standalone against an empty data directory/fully clean-slate.
2. Wait for the broker to become fully available. No operations need to be performed against it though.
3. Kill the broker with SIGKILL. I used `pkill -9 -f pulsar`.
4. Verify that the broker process is gone.
5. Attempt to start Pulsar again.
6. Observe that the broker never reaches a running state.

**Expected behavior**

Pulsar should start successfully after being brutally killed. This is an important simulation of e.g. a server power loss/kubernetes node abrupt termination event.

**Desktop (please complete the following information):**
- OS: MacOS 10.11

**Attachments**
- Attached are my full server output logs, containing two startup attempts: the first successful startup, then a line I wrote into the log by hand `KILLED HERE` indicating where I sent SIGKILL to the broker, followed by the second (failed) startup attempt.
- Also attached is a copy of my data directory after the crash.

[error.log](https://github.com/apache/pulsar/files/7078062/error.log)
[output.log](https://github.com/apache/pulsar/files/7078063/output.log)
[standalone.zip](https://github.com/apache/pulsar/files/7078064/standalone.zip)

Contributor guide

Open the contributing guide

Research direction

Reproduce the failure using the listed SIGKILL steps against an empty standalone data directory, then inspect error.log, output.log, and standalone.zip from the attachments. Done means standalone starts again after SIGKILL without deleting its data, or the issue documents a verified recovery workaround that preserves the data.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.