apache / apache/pulsar

Too many TCP Connections are in CLOSE_WAIT status freeze broker v 2.7.4

Open
#14,826 9 comments 0 reactions 0 assignees View on GitHub
lifecycle/stale Stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

**Describe the bug**
We saw strange behavior , Too many TCP Connections are in CLOSE_WAIT status in a Pulsar broker causing Disconnection Exceptions and Connection Already Close exceptions in pulsar clients as broker stopped accepting connections and clients start

The broker java process itself is up and running but looks freeze, as curl http ports such as curl broker metrics stop return anything.

It is only works when we restart the broker again

So it seems as connection pool issue or leak as it keeps alive specially we can some logs entries as below

[pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConnectionPool - [[id: 0x15a65d4f, L:/10.244.63.22:36068 - xxxxx:6651]] Connected to server Killed

And majority of the close_ wait connections between broker and bookies

Also i can see many occurrence of below exception:
Failed to initialize managed ledger: org.apache.bookkeeper.mledger.ManagedLedgerException$MetadataNotFoundException: Managed ledger not found
java.util.concurrent.CompletionException: java.util.NoSuchElementException: No value present
at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:884) ~[?:1.8.0_322]
at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:866) ~[?:1.8.0_322]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[?:1.8.0_322]
Caused by: java.util.NoSuchElementException: No value present

**To Reproduce**
The issue start to happen when K8 clients connecting our cluster (VM) and start producing messages with average volume 50k/s

**Expected behavior**
Broker should not freeze , and close the connections with clients when it received FIN ack

**Screenshots**
![image](https://user-images.githubusercontent.com/14996226/160095537-c0c217c8-4d14-47a4-94c9-9d47340e81c2.png)
**Desktop (please complete the following information):**
- OS: Ubuntu 18.04
- Java 8
- Pulsar running as systemd service
- 6 brokers
- 6 bookies
- 5 ZK

**Additional context**
No k8 deployment , and client accessing brokers DNS directly without Pulsar proxy.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the issue under the reported 50k/s producing load with six brokers, six bookies, and five ZooKeeper nodes. Inspect broker-to-client and broker-to-bookie connection handling alongside the ManagedLedgerException stack trace. Done means the broker remains responsive and closes client connections after FIN without accumulating CLOSE_WAIT connections.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.