apache / apache/pulsar

[Bug] Zookeeper OutOfMemoryError After Upgrading to Pulsar 3.3.0 with Zookeeper 3.9.2

Open
#23,348 3 comments 0 reactions 0 assignees View on GitHub
type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Read release policy

- [X] I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.

### Version

Pulsar version: 3.3.0
Zookeeper version: 3.9.2
Kubernetes environment: Helm chart deployment
Zookeeper resource configuration:
Request/limit: 6GB memory, 2 CPU
Heap settings: -Xms5632m -Xmx5632m
GC settings:

```
-XX:+UseG1GC
-XX:MaxGCPauseMillis=10
-XX:+ParallelRefProcEnabled
-XX:+UnlockExperimentalVMOptions
-XX:+DoEscapeAnalysis
-XX:+DisableExplicitGC
-XX:+ExitOnOutOfMemoryError
-XX:+PerfDisableSharedMem
```

### Minimal reproduce step

1. Upgrade Pulsar to version 3.3.0 and Zookeeper to 3.9.2.
2. Deploy the Zookeeper quorum using default GC and memory settings from the Helm chart.
3. Observe memory consumption and monitor for crashes after a few days of running.

### What did you expect to see?

Zookeeper should run without constantly increasing memory usage or exhausting resources.

### What did you see instead?

Error observed:

`java.lang.OutOfMemoryError: unable to create a native thread: possibly out of memory or process/resource limits reached.
`

### Anything else?

Additional Context:

After downgrading Zookeeper to version 3.2.2, the OOM issue stopped, and no pod restarts occurred.
Autorecovery pods running 3.3.0 are also encountering Java heap memory issues.
Reviewing the Pulsar 3.3.0 release notes and PIP-324 ([#22054](https://github.com/apache/pulsar/pull/22054)), I suspect changes to the Alpine base image could be affecting thread creation and memory management.

Possible Solution:

It may be necessary to modify the Dockerfile to increase the stack size by setting the PTHREAD_STACK_MIN environment variable:

`ENV PTHREAD_STACK_MIN 2097152`

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the Pulsar 3.3.0 release notes and PIP-324, then inspect the Dockerfile and Helm chart settings used for ZooKeeper. Reproduce the upgrade with the stated memory and GC configuration, monitor thread and heap usage, and confirm the change prevents the native-thread OutOfMemoryError and pod restarts.

Written by the indexing model from the issue text.

Assessment

Tech stack
helm, java, kubernetes
Domain
distributed-systems, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.