apache / apache/pulsar

[Bug]Pulsar 3.0.5 Topic service unavailable due to broken ledger in __change_events system topic

Open
#24,436 8 comments 0 reactions 0 assignees View on GitHub
type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before reporting

- [x] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Read release policy

- [x] I understand that [unsupported versions](https://pulsar.apache.org/contribute/release-policy/#supported-versions) don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.

### User environment

* **Broker Version**: 3.0.5

* **Deployment**: Kubernetes with Docker

* **Problem Description**

We have enabled the following features in our Pulsar cluster:

```yaml
systemTopicEnabled: "true"
topicLevelPoliciesEnabled: "true"

managedLedgerDefaultAckQuorum: "2"
managedLedgerDefaultEnsembleSize: "2"
managedLedgerDefaultWriteQuorum: "2"
```

### Issue Description

After configuring some topic policies, the cluster experienced several restarts or other operations over a period of time. We confirmed that no data was manually modified in BookKeeper during this period.

Subsequently, we observed "Failed to read entries" errors on the system topic `__change_events`. This issue then blocked the creation of both consumers and producers, rendering the topic service unavailable.

### Error messages

![Image](https://github.com/user-attachments/assets/b251fdaf-f4b0-4879-bb16-b91b8c82c99f)

Unfortunately, we do not have comprehensive logs from the exact time of the incident. However, this issue has occurred multiple times recently. We will ensure to collect more detailed logs if it recurs.

### Reproducing the issue

We don't have a stable way to reproduce this issue. Currently, we've observed that it **occurs with a higher probability in host restart scenarios.**

### Additional information

### **Workaround**

To resolve this, we followed these steps:

1. **Deleted the broken ledger** on the corresponding BookKeeper (BK) node.
2. After deletion, we might observe logs indicating "No such ledger exists on Metadata Server".
3. **Deleted the `/schemas/tenant/namespace/__change_events` entry** in ZooKeeper.
4. **Restarted the broker**.

After performing these steps, the cluster recovered.

### **Questions**

1. What could be the **root cause** for the ledger of the `__change_events` system topic becoming corrupted?
2. It seems **unreasonable** that a broken ledger in `__change_events` leads to the entire topic service becoming unavailable. Could there be an enhancement to detect such ledger corruption and **automatically reload/recreate** the necessary components to prevent service disruption?

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start with the broker handling of the __change_events system topic and trace the "Failed to read entries" path through BookKeeper and ZooKeeper. Try to reproduce the failure during host restarts with the listed Pulsar 3.0.5 settings, using detailed logs; done means the root cause is identified and the service no longer becomes unavailable when the ledger is broken.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, java, kubernetes
Domain
distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.