apache / apache/pulsar

[Bug][broker] Error reading entries due to LastConfirmedEntry is null until broker restart

Open
#23,654 7 comments 0 reactions 1 assignee Claimed by @BewareMyPower View on GitHub
type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Read release policy

- [X] I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.

### Version

OS: docker image apachepulsar/pulsar:3.3.2
Java: Amazon Corretto 21
Pulsar: 3.3.2 (client + server)

### Minimal reproduce step

Happens randomly during redeployment of a pulsar cluster - restarting all pulsar cluster components (brokers, bookies, zookeepers).

### What did you expect to see?

No mentioned error appear on brokers after restarting them.

### What did you see instead?

After brokers restart we start to see a lot of errors on brokers:
`[persistent://datalake/ingress/...-partition-... / sub] Error reading entries at 3904681:27 : LastConfirmedEntry is null when reading ledger 3904681, Read Type Normal - Retrying to read in 56.202 seconds"`
Retrying to read keeps happen until the next brokers restart.

At the same time we see growing backlog on that topic as the consumer can't read
![image](https://github.com/user-attachments/assets/1315d718-5dd5-4efc-84b8-bbc1a5a60bd4)

It happens on multipartition topic with the consumer from this class: org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers

When we detect it and restart all brokers once again then the backlog goes down (as you can see on the screenshot) and mentioned error goes away. So there's a workaround for that issue.

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.