apache / apache/opendal

new feature: let bindings configure outgoing HTTP requests

Open
#8,105 0 comments 1 reaction 0 assignees View on GitHub
bindings/java bindings/python enhancement releases-note/feat services/http
Dominant language
Rust
Stars
5.4k
Forks
825
Avg merge
1d 14m
Merged PRs (30d)
127

Description

### Feature Description

Rust users can influence the HTTP requests OpenDAL sends. They implement `HttpTransport` (RFC-7749) or pass their own `reqwest::Client`, then install it with `OperationContext::with_http_transport`. #4446 landed that, and #4365 removed OpenDAL's explicit `no_gzip`/`no_brotli`/`no_deflate` calls in order to "leave those decisions to the users".

Binding users have no equivalent. `HttpTransport` is a Rust trait, so Python and Java cannot supply a transport, and no operation option carries request headers. Every binding user is stuck with whatever the built-in transport sends.

I would like a supported way for a binding user to set request headers, and ideally to configure the transport more generally.

### Problem and Solution

The case I hit: Our gateway stores zstd-compressed objects and decompresses them on every read unless the client advertises the coding, in which case it streams the stored bytes through. Sending `Accept-Encoding: zstd` on reads moves that work off the gateway and shrinks the transfer.

The response half already works today. OpenDAL ships `reqwest` without codec features, so a `content-encoding: zstd` response is passed through undecoded, and the transport skips the consumed-vs-expected length check when `content-encoding` is present. I checked this against the published Python wheel and a stock Java build: both return the raw zstd frame, and `Metadata::content_encoding` reports the coding. Decoding in application code is fine, and matches #1225, which moved compression out of OpenDAL.

So the only missing piece is the request header, and it is missing solely because bindings cannot reach the transport.

Possible solution: An optional layer that adds headers to requests, exposed in the bindings:

```python
op = op.layer(opendal.layers.HttpHeaderLayer(
headers={"accept-encoding": "zstd"},
operations=["read"],
))
```

It needs no core API change. `Layer::apply_context` already intercepts the HTTP transport, and `ConcurrentLimitLayer`, `TracingLayer`, `HotpathLayer` and the metrics layers already wrap it that way. Filtering on the `Operation` request extension keeps a read-only header off `list` requests, which matters because S3 serves both over `GET`. Inserting only into vacant header slots keeps signed requests intact.

The part I would like guidance on: A layer like this can live outside the repo — `Layer`, `HttpTransport` and `HttpTransporter` are public — so Rust users can solve this themselves today, and that was suggested to me on #7894. But an out-of-tree layer is unreachable from Python and Java, because their `Layer` types wrap Rust implementations compiled into the extension module and the JNI library. Adding one means rebuilding and redistributing the wheels and jars.

So the question is not really "may I write a layer", it is: **how should bindings reach layers or transport configuration that OpenDAL does not ship?** If the answer is that they cannot, then this kind of feature has to live in the repo to be usable from a binding.

### Additional Context

Related:

- #7894 asks for customized headers as well, but with a different shape: per-call injection across all operator APIs, motivated by S3-compatible vendor quirks such as a backend requiring `Content-Length` on copy, which implies overriding headers OpenDAL sets. I am filing separately because my need is narrower — per-operator, insert-only, no overriding — and because it was pointed out on that issue that the two are not the same. I do not want to hijack that discussion.
- #4446 added the Rust-side ability to supply a client, and closed noting that header support for other bindings is a separate feature.
- #4365 "Remove reqwest related features" — the decision to leave codec choices to users.
- #1225 — compression belongs outside OpenDAL; this request keeps decoding in application code.

### Are you willing to contribute to the development of this feature?

- [x] Yes, I am willing to contribute to the development of this feature.

Contributor guide

Open the contributing guide

Research direction

Start with Layer::apply_context and the existing ConcurrentLimitLayer, TracingLayer, HotpathLayer, and metrics layers that wrap the HTTP transport. Then inspect how Layer types are exposed through the Python extension module and JNI library. Done means there is an agreed, supported way for binding users to reach scoped request-header or transport configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, python, rust
Domain
api, backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.