[Enhancement] Optimize Pulsar Java client zlib compression performance on Java 11+ by passing direct buffers
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Motivation
Here's an example of CompressionCodeZLib which has several opportunities for optimizations: https://github.com/apache/pulsar/blob/82237d3684fe506bcb6426b3b23f413422e6e4fb/pulsar-common/src/main/java/org/apache/pulsar/common/compression/CompressionCodecZLib.java#L60-L85
### Solution
The `java.util.zip.Deflater` class has contained methods for using `ByteBuffer` [input](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/zip/Deflater.html#setInput(java.nio.ByteBuffer)) and [output](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/zip/Deflater.html#deflate(java.nio.ByteBuffer,int)) since Java 11.
In the case of Java 11+, the code could be optimized.
Since the Pulsar Java client is Java 8+, using the `ByteBuffer` methods would require the use of reflection (unless a multi-release jar file is used with separate classes for Java 8 and Java 11). There's a reflection example in different situation in [BookKeeper's Java9IntHash class](https://github.com/apache/bookkeeper/blob/master/circe-checksum/src/main/java/com/scurrilous/circe/checksum/Java9IntHash.java).
Regarding performance on Java 11+, the first problem is that it's using a heap buffer for the compressed buffer. A direct buffer would be more optimal when using the ByteBuffer methods with Deflater.
For Netty ByteBuf input, it's possible to achieve zero copy in most cases by using Netty ByteBuf's `nioBuffer` method. It's notable that using `nioBuffer` method will cause copies when the Netty ByteBuf input is a CompositeByteBuf. Netty doesn't have a good way for zero copy of CompositeByteBuf input. In BookKeeper, there's a solution for checksum calculation in the https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/util/ByteBufVisitor.java class, which can visit all buffer parts to avoid extra copies. A similar solution would be applicable to compression.
### Alternatives
_No response_
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
Research direction
Start with pulsar-common/src/main/java/org/apache/pulsar/common/compression/CompressionCodecZLib.java and read the linked Deflater ByteBuffer APIs and BookKeeper Java9IntHash example. Compare Java 8 compatibility with the Java 11+ direct-buffer path, including Netty ByteBuf and CompositeByteBuf inputs. Done means compression preserves existing behavior while avoiding unnecessary copies on supported Java versions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, performance
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100