[2.8.2] error while create opSendMsg by batch message container - zstd overflow detected
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
**Describe the bug**
Just trying to upgrade from Pulsar 2.6.4 to 2.8.2, running into this exception very quickly on a standalone test (from scratch, so no backlog or anything).
```
WAR|11/145138.238 o.a.p.c.i.ProducerImpl@-client-io-68-10 [persistent:////] [] error while create opSendMsg by batch message container
java.lang.IllegalStateException: Overflow detected
at org.apache.pulsar.shade.io.airlift.compress.zstd.Util.checkState(Util.java:59)
at org.apache.pulsar.shade.io.airlift.compress.zstd.BitOutputStream.close(BitOutputStream.java:85)
at org.apache.pulsar.shade.io.airlift.compress.zstd.HuffmanCompressor.compressSingleStream(HuffmanCompressor.java:130)
at org.apache.pulsar.shade.io.airlift.compress.zstd.HuffmanCompressor.compress4streams(HuffmanCompressor.java:75)
at org.apache.pulsar.shade.io.airlift.compress.zstd.ZstdFrameCompressor.encodeLiterals(ZstdFrameCompressor.java:333)
at org.apache.pulsar.shade.io.airlift.compress.zstd.ZstdFrameCompressor.compressBlock(ZstdFrameCompressor.java:224)
at org.apache.pulsar.shade.io.airlift.compress.zstd.ZstdFrameCompressor.compressFrame(ZstdFrameCompressor.java:172)
at org.apache.pulsar.shade.io.airlift.compress.zstd.ZstdFrameCompressor.compress(ZstdFrameCompressor.java:145)
at org.apache.pulsar.shade.io.airlift.compress.zstd.ZStdRawCompressor.compress(ZStdRawCompressor.java:27)
at org.apache.pulsar.common.compression.CompressionCodecZstd.encode(CompressionCodecZstd.java:65)
at org.apache.pulsar.client.impl.BatchMessageContainerImpl.getCompressedBatchMetadataAndPayload(BatchMessageContainerImpl.java:135)
at org.apache.pulsar.client.impl.BatchMessageContainerImpl.createOpSendMsg(BatchMessageContainerImpl.java:189)
at org.apache.pulsar.client.impl.ProducerImpl.batchMessageAndSend(ProducerImpl.java:1796)
at org.apache.pulsar.client.impl.ProducerImpl.lambda$connectionOpened$13(ProducerImpl.java:1457)
at org.apache.pulsar.common.util.Runnables$CatchingAndLoggingRunnable.run(Runnables.java:53)
at org.apache.pulsar.shade.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
at org.apache.pulsar.shade.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:176)
at org.apache.pulsar.shade.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at org.apache.pulsar.shade.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
at org.apache.pulsar.shade.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
at org.apache.pulsar.shade.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at org.apache.pulsar.shade.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at org.apache.pulsar.shade.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:829)
```
and consumer received a erroneous message.
I changed nothing to the code and/or the configuration, so I was using batching and ZSTD compression with version 2.6.4.
**To Reproduce**
I'm not sure how to reproduce, here is the configuration for the producer:
```
builder
.producerName( producerName )
.topic( topic )
.enableBatching( true )
.batchingMaxMessages( 256 )
.batchingMaxPublishDelay( 150, TimeUnit.MILLISECONDS )
.blockIfQueueFull( true )
.maxPendingMessages( 1024 )
.maxPendingMessagesAcrossPartitions( 2 * 1024 )
.sendTimeout( 60000, TimeUnit.MILLISECONDS )
.compressionType( CompressionType.ZSTD );
```
The producer may be used by 16 threads concurrently. Individual messages are small.
I will try with a different compression if possible.
**Expected behavior**
Should not fail to compress the message.
**Desktop (please complete the following information):**
- OS: Ubuntu 20.04 / Java 11.0.13 2021-10-19
Contributor guide
Research direction
Start with the stack trace in client.impl.BatchMessageContainerImpl.getCompressedBatchMetadataAndPayload and createOpSendMsg, then follow CompressionCodecZstd into the shaded Zstandard compressor. Investigate the batching and concurrent producer configuration described in the report, and establish a reproducible case before changing behavior. Done means batched ZSTD sends no longer fail with overflow or deliver an erroneous message.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100