apache / apache/pulsar

[Bug] Consumer receive acknowledged messages from compacted topic after reconnection

Open
#23,494 0 comments 0 reactions 0 assignees View on GitHub
type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Read release policy

- [X] I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.

### Version

master

### Minimal reproduce step

1. producer send message
2. trigger compact
3. consume the compacted topic and acknowledge message
4. unload the topic and reconnect
5. re-consume the compacted topic
``` @Test
public void testIndividualAcknowledgeWithReconnection() throws Exception {
final String topicName = "persistent://my-property/use/my-ns/testIndividualAcknowledge" + UUID.randomUUID();
final String subName = "sub1";
final int numMessages = 10;

pulsarClient.newConsumer().topic(topicName).subscriptionName(subName)
.receiverQueueSize(1).readCompacted(true).subscribe().close();

Map expected = new HashMap<>();

@Cleanup
Producer producer = pulsarClient.newProducer(Schema.STRING)
.topic(topicName)
.enableBatching(false)
.messageRoutingMode(MessageRoutingMode.SinglePartition)
.create();

for (int i = 0; i < numMessages; i++) {
String key = "key" + new Random().nextInt(4);
String value = ("my-message-" + i);
producer.newMessage().key(key).value(value).send();
expected.put(key, value);
}
producer.flush();

// compact the topic
compact(topicName);

@Cleanup
Consumer consumer = pulsarClient.newConsumer(Schema.STRING).topic(topicName)
.subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
.subscriptionName(subName).receiverQueueSize(1).readCompacted(true).subscribe();

Map results = new HashMap<>();
while (true) {
Message message = consumer.receive(3, TimeUnit.SECONDS);
if (message == null) {
break;
}
results.put(message.getKey(), message.getValue());
consumer.acknowledge(message);
}
Awaitility.await().untilAsserted(() ->
assertEquals(admin.topics().getStats(topicName, true).getSubscriptions().get(subName).getMsgBacklog(),
numMessages - expected.size()));

// unload the topic
admin.topics().unload(topicName);

// wait the consumer reconnect
Awaitility.await().until(() -> admin.topics().getStats(topicName).getSubscriptions() != null);

// should not receive message
int count = 0;
while (true) {
Message message = consumer.receive(3, TimeUnit.SECONDS);
if (message == null) {
break;
}
count++;
results.put(message.getKey(), message.getValue());
consumer.acknowledge(message);
}
assertEquals(count, 0);

Awaitility.await().untilAsserted(() ->
assertEquals(admin.topics().getStats(topicName, true).getSubscriptions().get(subName).getMsgBacklog(),
numMessages - expected.size()));

assertEquals(results, expected);
}
```

### What did you expect to see?

In the step 5, we should receive no message

### What did you see instead?

Still receive the acknowledged message

### Anything else?

_No response_

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start with the supplied test method, testIndividualAcknowledgeWithReconnection, and run it against the master client and broker setup described in the issue. Reproduce the compact, acknowledge, unload, and reconnect sequence; done means the post-reconnection receive loop gets zero messages and both backlog and result assertions pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.