apache / apache/pulsar

[Bug] Consumer.batchReceive() with large batchSize setting times out when there are plenty of messages to receive

Open
#21,794 1 comment 0 reactions 0 assignees View on GitHub
type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Version

M1 macOS Sonoma + docker desktop 4.22.1, image: apachepulsar/pulsar:3.1.1, Java pulsar-client:3.1.1
also on linux/k8s

### Minimal reproduce step

using the following Java code and a local docker container running the apachepulsar/pulsar:3.1.1 image

calling `runTest(50000, 5000, 1000000)` consistently times out a lot (returning partial batches), mixed with a few quickly received full batches
calling `runTest(20000, 5000, 1000000)` never has a timeout/everything is received quickly

```
public static void runTest(
final int batchSize,
final long batchTimeoutMs,
final int totalMessages
) throws Exception {
final PulsarClient client = PulsarClient.builder()
.serviceUrl("pulsar://localhost:6650")
.ioThreads(8)
.connectionsPerBroker(8)
.build();

final Producer producer = client.newProducer()
.topic("persistent://public/default/test")
.compressionType(CompressionType.LZ4)
.messageRoutingMode(MessageRoutingMode.RoundRobinPartition)
.enableBatching(true)
.maxPendingMessages(2000)
.batchingMaxMessages(1000)
.blockIfQueueFull(true).create();

final Consumer consumer = client.newConsumer()
.topic("persistent://public/default/test")
.subscriptionName("test2")
.subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
.subscriptionType(SubscriptionType.Shared)
.ackTimeout(batchTimeoutMs + 60000, TimeUnit.MILLISECONDS)
.deadLetterPolicy(
DeadLetterPolicy.builder()
.maxRedeliverCount(10)
.build()
)
.batchReceivePolicy(
BatchReceivePolicy.builder()
.maxNumMessages(batchSize)
.timeout((int)batchTimeoutMs, TimeUnit.MILLISECONDS)
.build()
)
.receiverQueueSize(50000)
.subscribe();

for (int i = 0; i < totalMessages; i++) {
producer.sendAsync("item $i".getBytes());
}

int remainder = totalMessages;
int iteration = 1;
while (remainder > batchSize) { // compare to batchSize instead of zero since the last batch may not be full and would timeout, throwing off the test metric being observed
long start = System.currentTimeMillis();
final Messages batch = consumer.batchReceive();
System.out.println("batch " + iteration + " size = " + batch.size() + ", time = " + (System.currentTimeMillis() - start));
try { Thread.sleep(100); } catch (InterruptedException ignored) {}
consumer.acknowledge(batch);
remainder -= batch.size();
iteration += 1;
}

// clean out the remaining messages
while (remainder > 0) {
final Messages batch = consumer.batchReceive();
consumer.acknowledge(batch);
remainder -= batch.size();
}

client.close();
}
```

### What did you expect to see?

I expect to see no timeouts while the topic has more messages in the backlog than _batchSize_

I assume this something to do with some combination of `blockedSubscriptionOnUnackedMsgs`, `unackedMessages` and `availablePermits`. I read somewhere that if the number of `unackedMessages` is greater than 50000, flow to the consumer will stop, but the messages are ACKed very soon after being received, so I would expect flow to start again, and for the receive queue to fill up again before the timeout. It appears that I have to wait for a full timeout, or sometimes two, in order for the receive queue to fill up.

### What did you see instead?

timeouts occur intermittently, more frequently for larger _batchSize_ parameter values

### Anything else?

I want to use a large batchSize together with a long timeout threshold because this process writes message batches to file for long term storage, and I do not want a bunch of small files.

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start with the Consumer.batchReceive() path and investigate how blockedSubscriptionOnUnackedMsgs, unackedMessages, and availablePermits interact after acknowledgements. Run the provided Java reproduction with the apachepulsar/pulsar:3.1.1 image, comparing batchSize 20,000 and 50,000; done means large batches no longer intermittently wait for a timeout while messages remain available.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.