Standalone broker on MacOS often fail to start after workstation sleep or reboot
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
**Describe the bug**
Often, but not always, if I start Pulsar standalone on my workstation and then close/reopen my laptop lid or restart the computer, subsequent attempts to start the standalone broker fail. The JVM process keeps running and emits some errors in its logs (see attachment), but never reaches a connectible state.
Repeatedly restarting the broker usually causes this condition to go away. Sometimes that doesn't work, and I have to remove all broker data files (i.e. reinstall Pulsar) to make it start.
This only seems to happen when publishes have been recorded to a persistent topic.
This does not happen when using Pulsar Standalone in Docker, or on my Linux laptop. It appears to be MacOS specific.
**To Reproduce**
1. `brew install apache-pulsar`
2. `brew services start apache-pulsar`
3. Verify that Pulsar is connectible on localhost:6650.
4. Publish some messages to a persistent topic such that the message data and the topic itself are not automatically deleted (i.e. at least one ledger is created and persists).
5. Close laptop lid or otherwise induce sleep.
6. Reopen laptop and attempt to connect to Pulsar; it will sometimes fail.
7. `brew services restart apache-pulsar` and attempt to connect to Pulsar; it will sometimes fail.
**Expected behavior**
1. Sleep/resume should not break Pulsar. It should either be available after resumption, or the service/JVM process should fully crash so that it reports not-running if it cannot handle connection requests after resumption.
2. Restarting the service should result in Pulsar either being connectible or not having a JVM process. If Pulsar Standalone can't come up in a connectible state, its process shouldn't hang around (if the process stays around, service managers think it's still running and thus won't report a failure or auto-restart it).
**Desktop (please complete the following information):**
- OS: MacOS monterey, intel.
- Pulsar standalone 2.9.2 and 2.9.1 both exhibit this issue.
**Additional context**
`brew services` is likely not the problem. It doesn't do anything special, and Pulsar does sometimes work when restarted via `brew services`.
Attached is a copy of my logs from a broker start attempt that did not become connectible after 5min.
[nostart.log](https://github.com/apache/pulsar/files/8597454/nostart.log)
Contributor guide
Research direction
Start with the attached nostart.log and reproduce the seven-step sleep/resume sequence using Pulsar standalone started by `brew services`. Trace why the JVM remains running but is not connectible after persistent-topic data exists, then verify that resume and service restart either restore connectivity or terminate the process as expected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, macos
- Domain
- backend, distributed-systems, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100