apache / apache/pulsar

[2.8.2] error while create opSendMsg by batch message container - zstd overflow detected

Open
#13,712 5 comments 0 reactions 0 assignees View on GitHub
lifecycle/stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

**Describe the bug**
Just trying to upgrade from Pulsar 2.6.4 to 2.8.2, running into this exception very quickly on a standalone test (from scratch, so no backlog or anything).

```
WAR|11/145138.238 o.a.p.c.i.ProducerImpl@-client-io-68-10 [persistent:////] [] error while create opSendMsg by batch message container
java.lang.IllegalStateException: Overflow detected
at org.apache.pulsar.shade.io.airlift.compress.zstd.Util.checkState(Util.java:59)
at org.apache.pulsar.shade.io.airlift.compress.zstd.BitOutputStream.close(BitOutputStream.java:85)
at org.apache.pulsar.shade.io.airlift.compress.zstd.HuffmanCompressor.compressSingleStream(HuffmanCompressor.java:130)
at org.apache.pulsar.shade.io.airlift.compress.zstd.HuffmanCompressor.compress4streams(HuffmanCompressor.java:75)
at org.apache.pulsar.shade.io.airlift.compress.zstd.ZstdFrameCompressor.encodeLiterals(ZstdFrameCompressor.java:333)
at org.apache.pulsar.shade.io.airlift.compress.zstd.ZstdFrameCompressor.compressBlock(ZstdFrameCompressor.java:224)
at org.apache.pulsar.shade.io.airlift.compress.zstd.ZstdFrameCompressor.compressFrame(ZstdFrameCompressor.java:172)
at org.apache.pulsar.shade.io.airlift.compress.zstd.ZstdFrameCompressor.compress(ZstdFrameCompressor.java:145)
at org.apache.pulsar.shade.io.airlift.compress.zstd.ZStdRawCompressor.compress(ZStdRawCompressor.java:27)
at org.apache.pulsar.common.compression.CompressionCodecZstd.encode(CompressionCodecZstd.java:65)
at org.apache.pulsar.client.impl.BatchMessageContainerImpl.getCompressedBatchMetadataAndPayload(BatchMessageContainerImpl.java:135)
at org.apache.pulsar.client.impl.BatchMessageContainerImpl.createOpSendMsg(BatchMessageContainerImpl.java:189)
at org.apache.pulsar.client.impl.ProducerImpl.batchMessageAndSend(ProducerImpl.java:1796)
at org.apache.pulsar.client.impl.ProducerImpl.lambda$connectionOpened$13(ProducerImpl.java:1457)
at org.apache.pulsar.common.util.Runnables$CatchingAndLoggingRunnable.run(Runnables.java:53)
at org.apache.pulsar.shade.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
at org.apache.pulsar.shade.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:176)
at org.apache.pulsar.shade.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at org.apache.pulsar.shade.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
at org.apache.pulsar.shade.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
at org.apache.pulsar.shade.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at org.apache.pulsar.shade.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at org.apache.pulsar.shade.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:829)

```
and consumer received a erroneous message.
I changed nothing to the code and/or the configuration, so I was using batching and ZSTD compression with version 2.6.4.

**To Reproduce**
I'm not sure how to reproduce, here is the configuration for the producer:

```
builder
.producerName( producerName )
.topic( topic )
.enableBatching( true )
.batchingMaxMessages( 256 )
.batchingMaxPublishDelay( 150, TimeUnit.MILLISECONDS )
.blockIfQueueFull( true )
.maxPendingMessages( 1024 )
.maxPendingMessagesAcrossPartitions( 2 * 1024 )
.sendTimeout( 60000, TimeUnit.MILLISECONDS )
.compressionType( CompressionType.ZSTD );
```

The producer may be used by 16 threads concurrently. Individual messages are small.
I will try with a different compression if possible.

**Expected behavior**
Should not fail to compress the message.

**Desktop (please complete the following information):**
- OS: Ubuntu 20.04 / Java 11.0.13 2021-10-19

Contributor guide

Open the contributing guide

Research direction

Start with the stack trace in client.impl.BatchMessageContainerImpl.getCompressedBatchMetadataAndPayload and createOpSendMsg, then follow CompressionCodecZstd into the shaded Zstandard compressor. Investigate the batching and concurrent producer configuration described in the report, and establish a reproducible case before changing behavior. Done means batched ZSTD sends no longer fail with overflow or deliver an erroneous message.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.