apache / apache/opendal

new feature: allow users to inject customized HTTP header

Open
#7,894 3 comments 2 reactions 0 assignees View on GitHub
enhancement releases-note/feat services/http
Dominant language
Rust
Stars
5.4k
Forks
825
Avg merge
1d 14m
Merged PRs (30d)
127

Description

### Feature Description

Hi team, there're a lot of storage backends which declare themself to be S3 compatible, but they have slight differences.
One concrete example is https://github.com/apache/opendal/issues/7821, netapp S3 compatible object storage requires content-length in copy request, otherwise the request will get rejected.

OpenDAL, from my perspective, should focus on the well-documented S3 spec, instead of adding undocumented logic for different vendor impls.
- Each vendors' implementation could conflict with each other
- Support different types of S3 backend in OpenDAL core increases maintenance overhead
+ For example, netapp S3 API is not open source, with undocumented request format, no docker image provided, I don't even know how can we test and maintain

So I'm wondering if it's better to allow users to inject their own header.
- User-side should have little overhead to maintain a small list of headers
- Vendors could have their customized headers for performance (the object storage I'm working does so)

This feature request is not specific to S3 backend, but should be general enough for all backends.

### Problem and Solution

For all the operator APIs, we allow users to inject their own headers, which will be propagated to HTTP requests.

### Additional Context

See original PR and thread for details at https://github.com/apache/opendal/pull/7884#issuecomment-4926332061

### Are you willing to contribute to the development of this feature?

- [x] Yes, I am willing to contribute to the development of this feature.

Contributor guide

Open the contributing guide

Research direction

Start with the original PR #7884 and its linked discussion, then trace how the operator APIs construct HTTP requests across storage backends. Define how user-provided headers should be supplied and propagated generally, and consider how the behavior can be verified for all affected operator APIs without adding vendor-specific logic.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.