apache / apache/pulsar

[Bug] Consumer might not be able to consume messages after failover to a new cluster

Open
#25,291 0 comments 0 reactions 1 assignee Claimed by @BewareMyPower View on GitHub
type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before reporting

- [x] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Read release policy

- [x] I understand that [unsupported versions](https://pulsar.apache.org/contribute/release-policy/#supported-versions) don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.

### User environment

master branch

### Issue Description

With `AutoClusterFailover`, a consumer is able to switch from cluster A to cluster B on the same topic.

https://github.com/apache/pulsar/blob/629ddebeec08438536a6b63f413c11caaf445b1f/pulsar-client/src/main/java/org/apache/pulsar/client/impl/AutoClusterFailover.java#L172-L173

The client will update the service URL and recreate the lookup service. However, the consumer's internal states are not cleared, including:
- The ACK grouping tracker, which contains some acknowledged message IDs
- `lastDequedMessageId`, which represents the message ID of last message received

These message IDs all come from **Cluster A**.

Next time, when the consumer established connection to Cluster B, it will:
- Filter out duplicated messages via `AcknowledgmentsGroupingTracker#isDuplicate`. If the acknowledged message ID is cumulative, it will compare the newly **received message's ID from Cluster B** with **the previous acknowledged message ID from Cluster A**
- Set `startMessageId` to the **last message ID from Cluster A**, this message ID will be carried in **the Subscribe request to the topic on Cluster B**

### Error messages

```text
n/a
```

### Reproducing the issue

I did't try to reproduce it with Java client for now, but it can be reproduced with C++ client like: https://github.com/apache/pulsar-client-cpp/pull/541/changes/c6de067e3795b58eed558c03795e385fc8ee16a5#diff-36936d31d0cbc6547ff0eea6b2bc79bbffdc4de6d63223dd0b4e527e369059d1

### Additional information

_No response_

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.