apache / apache/arrow-rs-object-store
S3 + HTTP/2: "duplicate" header
- Dominant language
- Rust
- Stars
- 322
- Forks
- 212
- Avg merge
- 5d 2h
- Merged PRs (30d)
- 10
Description
**Describe the bug**
The bug occurs when S3 signatures are used in combination with HTTP/2. This (IIRC) is currently not possible with AWS S3, but some other vendors or custom implementation may run into this.
The combination produces the special `:authority:` HTTP/2 pseudo-header as well as the `Host` header. Some server or middleware implementation don't like that, e.g. Nginx will return a "bad request" and emit the log (here the host was `localhost:9999`:
```text
client sent duplicate host header: "host: localhost:9999", previous value: "host: localhost:9999" while reading client request headers, client: 127.0.0.1, server: , host: "localhost:9999"
```
Note that the header is not really duplicate. It's likely that Nginx internally renames `:authority` to `Host` and then trips over it. Also see https://trac.nginx.org/nginx/ticket/2268 .
**To Reproduce**
```rust
use object_store::{
aws::AmazonS3Builder,
ClientOptions,
};
let store = AmazonS3Builder::new()
.with_client_options(
ClientOptions::new()
.with_http2_only()
)
.with_region(...)
.with_access_key_id(...)
.with_secret_access_key(...)
.build()
.unwrap();
store.get(...).await.unwrap();
```
**Expected behavior**
**Additional context**
I think the culprit is this bit here:
https://github.com/apache/arrow-rs/blob/97ae9d778b5f1d1fcc0c7beb91b2b1a6ed741194/object_store/src/aws/credential.rs#L157-L159
Of course one could argue that `reqwest` or `h2` should de-duplicate the headers properly.
Contributor guide
Research direction
Start with the HTTP/2 reproduction using AmazonS3Builder and ClientOptions.with_http2_only(), then inspect object_store/src/aws/credential.rs around the linked lines. Done means the S3-signed request can pass through the affected server or middleware without triggering a duplicate Host-header error; no test file is named in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, rust
- Domain
- cloud, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100