apache / apache/pulsar

[Bug] Broker became irresponsive due to too many open files error

Open
#22,894 3 comments 0 reactions 0 assignees View on GitHub
type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Read release policy

- [X] I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.

### Version

`>= 2.10`

### Minimal reproduce step

Suddenly broker log shows below error and connected producers started seeing timeout for published messages
```
04:05:33.877 [pulsar-stats-updater-24-1] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [pulsar/us-east1/broker1-use1:4080/persistent/topic1-1]
Closing inactive ledger, last-add entry 0
04:06:33.877 [pulsar-stats-updater-24-1] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [pulsar/us-east1/broker1-use1:4080/persistent/topic1-0]
Closing inactive ledger, last-add entry 0
04:06:33.877 [pulsar-stats-updater-24-1] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [pulsar/us-east1/broker1-use1:4080/persistent/topic1-2]
Closing inactive ledger, last-add entry 0
04:06:33.877 [pulsar-stats-updater-24-1] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [pulsar/us-east1/broker1-use1:4080/persistent/topic1-1]
Closing inactive ledger, last-add entry 0
04:07:26.759 [pulsar-acceptor-23-1] WARN io.netty.channel.DefaultChannelPipeline - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last han
dler in the pipeline did not handle the exception.
io.netty.channel.unix.Errors$NativeIoException: accept(..) failed: Too many open files
04:07:27.760 [pulsar-acceptor-23-1] WARN io.netty.channel.DefaultChannelPipeline - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.channel.unix.Errors$NativeIoException: accept(..) failed: Too many open files
04:07:28.760 [pulsar-acceptor-23-1] WARN io.netty.channel.DefaultChannelPipeline - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.channel.unix.Errors$NativeIoException: accept(..) failed: Too many open files
04:07:29.760 [pulsar-acceptor-23-1] WARN io.netty.channel.DefaultChannelPipeline - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.channel.unix.Errors$NativeIoException: accept(..) failed: Too many open files
04:07:30.761 [pulsar-acceptor-23-1] WARN io.netty.channel.DefaultChannelPipeline - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.channel.unix.Errors$NativeIoException: accept(..) failed: Too many open files
04:07:31.761 [pulsar-acceptor-23-1] WARN io.netty.channel.DefaultChannelPipeline - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.channel.unix.Errors$NativeIoException: accept(..) failed: Too many open files
04:07:32.761 [pulsar-acceptor-23-1] WARN io.netty.channel.DefaultChannelPipeline - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.channel.unix.Errors$NativeIoException: accept(..) failed: Too many open files
04:07:33.761 [pulsar-acceptor-23-1] WARN io.netty.channel.DefaultChannelPipeline - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.channel.unix.Errors$NativeIoException: accept(..) failed: Too many open files
```

Listing open files shows that large number of connections are in `CLOSE_WAIT` state but we don't see any other additional information when broker goes in that state.

### What did you expect to see?

Broker should not go in such an irresponsive state.

### What did you see instead?

Client started seeing publish timeout.

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!
**I am not willing to create PR due to duplicate efforts by multiple contributors and avoid wasting time.**

Contributor guide

Open the contributing guide

Research direction

Start with the broker logs around the pulsar-acceptor Netty pipeline and the ManagedLedgerImpl entries, then inspect the reported CLOSE_WAIT connections and open-file usage. Reproduce the broker's accept(..) failure if possible; done means identifying the cause and demonstrating that the broker no longer becomes unresponsive or causes publish timeouts.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.