[Bug] Consumer receive acknowledged messages from compacted topic after reconnection
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Read release policy
- [X] I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.
### Version
master
### Minimal reproduce step
1. producer send message
2. trigger compact
3. consume the compacted topic and acknowledge message
4. unload the topic and reconnect
5. re-consume the compacted topic
``` @Test
public void testIndividualAcknowledgeWithReconnection() throws Exception {
final String topicName = "persistent://my-property/use/my-ns/testIndividualAcknowledge" + UUID.randomUUID();
final String subName = "sub1";
final int numMessages = 10;
pulsarClient.newConsumer().topic(topicName).subscriptionName(subName)
.receiverQueueSize(1).readCompacted(true).subscribe().close();
Map expected = new HashMap<>();
@Cleanup
Producer producer = pulsarClient.newProducer(Schema.STRING)
.topic(topicName)
.enableBatching(false)
.messageRoutingMode(MessageRoutingMode.SinglePartition)
.create();
for (int i = 0; i < numMessages; i++) {
String key = "key" + new Random().nextInt(4);
String value = ("my-message-" + i);
producer.newMessage().key(key).value(value).send();
expected.put(key, value);
}
producer.flush();
// compact the topic
compact(topicName);
@Cleanup
Consumer consumer = pulsarClient.newConsumer(Schema.STRING).topic(topicName)
.subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
.subscriptionName(subName).receiverQueueSize(1).readCompacted(true).subscribe();
Map results = new HashMap<>();
while (true) {
Message message = consumer.receive(3, TimeUnit.SECONDS);
if (message == null) {
break;
}
results.put(message.getKey(), message.getValue());
consumer.acknowledge(message);
}
Awaitility.await().untilAsserted(() ->
assertEquals(admin.topics().getStats(topicName, true).getSubscriptions().get(subName).getMsgBacklog(),
numMessages - expected.size()));
// unload the topic
admin.topics().unload(topicName);
// wait the consumer reconnect
Awaitility.await().until(() -> admin.topics().getStats(topicName).getSubscriptions() != null);
// should not receive message
int count = 0;
while (true) {
Message message = consumer.receive(3, TimeUnit.SECONDS);
if (message == null) {
break;
}
count++;
results.put(message.getKey(), message.getValue());
consumer.acknowledge(message);
}
assertEquals(count, 0);
Awaitility.await().untilAsserted(() ->
assertEquals(admin.topics().getStats(topicName, true).getSubscriptions().get(subName).getMsgBacklog(),
numMessages - expected.size()));
assertEquals(results, expected);
}
```
### What did you expect to see?
In the step 5, we should receive no message
### What did you see instead?
Still receive the acknowledged message
### Anything else?
_No response_
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
Research direction
Start with the supplied test method, testIndividualAcknowledgeWithReconnection, and run it against the master client and broker setup described in the issue. Reproduce the compact, acknowledge, unload, and reconnect sequence; done means the post-reconnection receive loop gets zero messages and both backlog and result assertions pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100