apache / apache/pulsar

[Enhancement] Optimize Pulsar Java client zlib compression performance on Java 11+ by passing direct buffers

Open
#23,586 1 comment 0 reactions 0 assignees View on GitHub
type/enhancement
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Motivation

Here's an example of CompressionCodeZLib which has several opportunities for optimizations: https://github.com/apache/pulsar/blob/82237d3684fe506bcb6426b3b23f413422e6e4fb/pulsar-common/src/main/java/org/apache/pulsar/common/compression/CompressionCodecZLib.java#L60-L85

### Solution

The `java.util.zip.Deflater` class has contained methods for using `ByteBuffer` [input](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/zip/Deflater.html#setInput(java.nio.ByteBuffer)) and [output](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/zip/Deflater.html#deflate(java.nio.ByteBuffer,int)) since Java 11.

In the case of Java 11+, the code could be optimized.
Since the Pulsar Java client is Java 8+, using the `ByteBuffer` methods would require the use of reflection (unless a multi-release jar file is used with separate classes for Java 8 and Java 11). There's a reflection example in different situation in [BookKeeper's Java9IntHash class](https://github.com/apache/bookkeeper/blob/master/circe-checksum/src/main/java/com/scurrilous/circe/checksum/Java9IntHash.java).

Regarding performance on Java 11+, the first problem is that it's using a heap buffer for the compressed buffer. A direct buffer would be more optimal when using the ByteBuffer methods with Deflater.
For Netty ByteBuf input, it's possible to achieve zero copy in most cases by using Netty ByteBuf's `nioBuffer` method. It's notable that using `nioBuffer` method will cause copies when the Netty ByteBuf input is a CompositeByteBuf. Netty doesn't have a good way for zero copy of CompositeByteBuf input. In BookKeeper, there's a solution for checksum calculation in the https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/util/ByteBufVisitor.java class, which can visit all buffer parts to avoid extra copies. A similar solution would be applicable to compression.

### Alternatives

_No response_

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start with pulsar-common/src/main/java/org/apache/pulsar/common/compression/CompressionCodecZLib.java and read the linked Deflater ByteBuffer APIs and BookKeeper Java9IntHash example. Compare Java 8 compatibility with the Java 11+ direct-buffer path, including Netty ByteBuf and CompositeByteBuf inputs. Done means compression preserves existing behavior while avoiding unnecessary copies on supported Java versions.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, performance
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.