apache / apache/pulsar

[Bug] Standalone broker does not become responsive after start.

Open
#20,875 6 comments 0 reactions 0 assignees View on GitHub
Stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Version

Pulsar 2.10.3.

Docker v4.21.1
Pulsar is the only container running. Docker has 10 CPUs, 24GB of RAM, and 4GB of swap. The host machine is not under memory or CPU pressure (running a terminal and Docker, no other programs, activity monitor reports idle).

Docker engine: 24.0.2

Config:
```
{
"builder": {
"gc": {
"defaultKeepStorage": "180GB",
"enabled": true
}
},
"experimental": true,
"features": {
"buildkit": true
}
}
```

MacOS 12.6.7

ARM/M1 processor.

### Minimal reproduce step

- Run this Docker command:
```
docker run \
--rm \
--name chariot_local_pulsar \
-it \
-p 6650:6650 \
-p 8080:8080 \
--cap-add=SYS_PTRACE \
--platform linux/x86_64 \
apachepulsar/pulsar:2.10.3 \
bin/pulsar standalone -nss -nfw
```

### What did you expect to see?

Within a few minutes, a broker that I can use for admin API and messaging.

### What did you see instead?

This bug is **intermittent**. Usually, things work fine.

But sometimes (about 1 or 2 out of every 5 times), the broker never starts up; it writes the attached logs and then stops emitting any log info (no activity for 30min). I can not connect to the management API or the messaging port and can not perform any operations. No attempts to connect to either port cause new log output to occur.

### Anything else?

I have observed similar problems on Pulsar 3.0, with different logs.

Restarting the computer, or resetting docker (`docker system prune --all --force`) doesn't seem to reduce or increase the likelihood of this occurring. The very first broker start seems about as likely to fail as any subsequent restart, whether or not the docker daemon was restarted or the image was re-pulled.

If I `exec` into the container and attempt to capture a heap dump, I get the following error:

```
I have no name!@2f8835389778:/pulsar$ jmap -dump:live,format=b,file=dump.hprof 1
Exception in thread "main" com.sun.tools.attach.AttachNotSupportedException: Unable to open socket file /proc/1/root/tmp/.java_pid1: target process 1 doesn't respond within 10500ms or HotSpot VM not loaded
at jdk.attach/sun.tools.attach.VirtualMachineImpl.(VirtualMachineImpl.java:100)
at jdk.attach/sun.tools.attach.AttachProviderImpl.attachVirtualMachine(AttachProviderImpl.java:58)
at jdk.attach/com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:207)
at jdk.jcmd/sun.tools.jmap.JMap.executeCommandForPid(JMap.java:128)
at jdk.jcmd/sun.tools.jmap.JMap.dump(JMap.java:208)
at jdk.jcmd/sun.tools.jmap.JMap.main(JMap.java:114)
```

`top` doesn't report resource exhaustion in the container:
```
top - 16:33:00 up 1:02, 0 users, load average: 1.00, 1.01, 1.00
Tasks: 3 total, 0 running, 3 sleeping, 0 stopped, 0 zombie
%Cpu(s): 10.1 us, 0.0 sy, 0.0 ni, 89.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 24010.6 total, 19671.0 free, 1454.2 used, 2885.4 buff/cache
MiB Swap: 4096.0 total, 4096.0 free, 0.0 used. 21960.3 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 10000 20 0 9907.3m 883056 35864 S 101.0 3.6 19:22.82 java
423 10000 20 0 149248 13184 4992 S 0.0 0.1 0:00.07 bash
457 10000 0 0 150872 11024 4876 0 0.0 0.0 0:00.00 top
```

`ps` output:
```
UID PID PPID C STIME TTY TIME CMD
10000 1 0 99 16:14 pts/0 00:21:11 /usr/bin/qemu-x86_64 /usr/lib/jvm/java-11-openjdk-amd64/bin/java /usr/lib/jvm/java-11-openjdk-amd64/bin/java -Dlog4j.shutdo
10000 423 0 0 16:24 pts/1 00:00:00 /usr/bin/qemu-x86_64 /bin/bash /bin/bash
10000 529 423 0 15:30 ? 00:00:00 ps -ef
```

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start with the provided `docker run` command and the `bin/pulsar standalone -nss -nfw` entry point, comparing logs from successful and failed starts on the stated ARM/M1 environment. Investigate why the intermittent process stops responding, and consider the issue done when both the admin API and messaging port become reliably available after startup.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, java
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.