apache / apache/pekko-http

Allow plugin support for custom compression/decompression (e.g. zstd)

Open
#997 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
196
Forks
55
Avg merge
4d 52m
Merged PRs (30d)
74

Description

## Feature request: Plug-in support for custom compression (e.g., zstd) in Pekko HTTP coding subsystem

### Context
Currently, Pekko HTTP provides built-in support for gzip (and deflate) compression via the `Coder` abstraction in `org.apache.pekko.http.scaladsl.coding`. This is used via `encodeResponse`, `decodeRequest`, and related directives, and the negotiation machinery is well-established. There is frequent interest (e.g., for modern formats like zstd) in supporting custom content encoding algorithms. Users may want to plug in their own encoders/decoders without modifying Pekko HTTP core code.

There is #860 which is about adding built-in support for zstd but it might be worth considering allowing the addition of custom compression/decompression so users can add support themselves.

### Analysis
After reviewing the codebase ([scaladsl/coding](https://github.com/apache/pekko-http/tree/main/http/src/main/scala/org/apache/pekko/http/scaladsl/coding/)), here's what enables and what currently blocks custom compressions:

#### What works:
- `Coder` is a marker trait (`trait Coder extends Encoder with Decoder`). Users can implement it today to support new encodings, e.g., zstd.
- `HttpEncoding.custom(...)` allows defining new encoding tokens, such as `zstd`.
- Directives (`encodeResponseWith`, `decodeRequestWith`) accept arbitrary `Encoder`/`Decoder`s, so user-provided implementations can already be injected explicitly per-route.

#### Current barriers:
1. **Compressor and StreamDecoder are Internal/Deprecated:**
- `Compressor` (needed for encoding) and `StreamDecoder` (for decoding) are `@InternalApi` and `@deprecated`. Implementing `Encoder`/`Decoder` needs usage of these, but they're not public API and may change or disappear, so user-providers can't rely on them.
2. **Java API is closed:**
- The Java-side equivalent (`org.apache.pekko.http.javadsl.coding.Coder`) is a closed `enum`—Java users cannot add custom encoders at all.
3. **Default coders are fixed:**
- `Coders.DefaultCoders` is always `[Gzip, Deflate, NoCoding]`. There is no mechanism for registering a custom coder with the negotiation machinery globally. So, while you can use your custom coder per-route with the `With` directives, it won't be automatically used with default directives.

#### Example: How it could work (after stabilising APIs)
```scala
object ZstdCoder extends Coder with StreamDecoder {
val encoding: HttpEncoding = HttpEncoding.custom("zstd")
def messageFilter = Encoder.DefaultFilter
private[this] def newCompressor = new ZstdCompressor() // if Compressor is public
def newDecompressorStage(maxBytesPerChunk: Int) = () => new ZstdDecompressorStage(maxBytesPerChunk)
}
// Usage:
encodeResponseWith(ZstdCoder, Coders.Gzip, Coders.Deflate, Coders.NoCoding) { ... }
decodeRequestWith(ZstdCoder, Coders.Gzip, Coders.Deflate) { ... }
```

### Proposed Enhancements
1. **Promote Compressor and StreamDecoder to Public API:**
- Remove `@InternalApi` and `@deprecated` from these types
- Alternatively, provide a new public API for plugging in custom encoder/decoder flows
2. **Open up Java API for Custom Coders:**
- Replace or supplement the hardcoded `Coder` enum with an interface/factory-based mechanism in the Java DSL
3. **Make DefaultCoders Configurable or Injectable:**
- Allow applications to register additional coders (e.g., via ActorSystem config or a registration method) for use by the default directives/negotiator
4. **(Optional) Add zstd as a built-in or contrib module:**
- To showcase extensibility and provide instant value
5. **(Docs) Add a guide for implementing and registering custom coders**

### Impact
- This would unlock modern and/or domain-specific compression for Pekko HTTP users
- Immediate benefit for projects that need zstd or other algorithms (Brotli, LZ4, etc.)
- Makes Pekko HTTP encoding/decoding as extensible as its routing layers

**If maintainers agree, I'm happy to help with API stabilisation PRs or further design discussion!**

---
Origin: https://github.com/apache/pekko-http (discussion/analysis by pjfanning)

Contributor guide

Open the contributing guide

Research direction

Read http/src/main/scala/org/apache/pekko/http/scaladsl/coding/, focusing on Coder, Compressor, StreamDecoder, and Coders.DefaultCoders, then inspect the Java coding API. Define the supported extension points and negotiation behavior for custom coders in both DSLs. Done requires an agreed API design with implementation tests and documentation for registration and use.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, scala
Domain
backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.