S3 async clients (CRT and Netty) always use CONNECT tunneling with a proxy, even for plaintext http:// endpoints
- Lingua principale
- Java
- Stelle
- 2.6k
- Fork
- 1k
- Merge medio
- 2g 9h
- PR unite (30g)
- 51
Descrizione
### Describe the bug
When an HTTP proxy is configured, the asynchronous S3 clients (`S3AsyncClient.crtBuilder()` and `S3AsyncClient.builder()`) always establish the proxy connection with a `CONNECT` tunnel, even when `endpointOverride` uses the plaintext `http://` scheme. The synchronous `S3Client` backed by `ApacheHttpClient` correctly uses forwarding mode for the same configuration.
Many corporate proxies deliberately disallow the `CONNECT` method (tunnels defeat traffic inspection and can be used to bypass egress controls) and only support forwarding. Against such a proxy the async S3 clients cannot be used at all, while the sync client works.
Note that the emitted request is self-contradictory: the client sends `CONNECT host:80` — it knows the target is plaintext (hence port 80) yet still negotiates a tunnel as if TLS were in use.
### Regression Issue
- [ ] Select this option if this issue appears to be a regression.
Not a regression — reproduced identically on 2.33.0 and 2.54.4.
### Expected Behavior
With `endpointOverride = http://` and a proxy configured, the client should use forwarding mode and send an absolute-form request, matching both `curl -x http:// http:///` and the SDK's own `ApacheHttpClient`:
```
GET http:///?list-type=2 HTTP/1.1
```
### Current Behavior
The client sends:
```
CONNECT :80 HTTP/1.1
```
Against a proxy that rejects `CONNECT`, the request fails with:
```
software.amazon.awssdk.core.exception.SdkClientException:
Failed to send the request: Proxy-based connection establishment failed because the CONNECT call failed
```
**Behavior matrix observed (SDK 2.54.4, aws-crt 0.48.4)** — identical configuration, only the HTTP client differs:
| HTTP client | `http://` endpoint | `https://` endpoint |
|---|---|---|
| `ApacheHttpClient` (sync `S3Client`) | `GET http://host/...` :white_check_mark: | `CONNECT host:443` :white_check_mark: |
| CRT (`S3AsyncClient.crtBuilder()`) | `CONNECT host:80` :x: | `CONNECT host:443` :white_check_mark: |
| Netty (`S3AsyncClient.builder()`) | `CONNECT host:80` :x: | `CONNECT host:443` :white_check_mark: |
The same CRT client is also inconsistent with itself depending on whether a proxy is configured:
| Scenario | endpoint | First bytes sent |
|---|---|---|
| CRT, **no proxy** | `http://` | `GET /?list-type=2 HTTP/1.1` (plaintext) :white_check_mark: |
| CRT, **no proxy** | `https://` | TLS ClientHello :white_check_mark: |
| CRT, **with proxy** | `http://` | `CONNECT host:80` :x: |
So the CRT client honours the endpoint scheme when connecting directly, but stops honouring it once a proxy is involved.
### Reproduction Steps
**1. Start a minimal proxy stub** that prints the first line it receives and closes the connection:
```python
# stub_proxy.py
import socket, sys, threading
port = int(sys.argv[1])
s = socket.socket(); s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind(("127.0.0.1", port)); s.listen(8)
def handle(c):
data = c.recv(4096)
line = data.split(b"\r\n")[0].decode("latin1") if data else "(empty)"
if data[:1] == b"\x16": line = ""
print(f"port {port} FIRST LINE: {line}", flush=True)
c.close()
while True:
c, _ = s.accept(); threading.Thread(target=handle, args=(c,), daemon=True).start()
```
```bash
python3 stub_proxy.py 8888 &
python3 stub_proxy.py 8889 &
python3 stub_proxy.py 8890 &
```
**2. Run each client against a plaintext endpoint:**
```java
import java.net.URI;
import software.amazon.awssdk.auth.credentials.*;
import software.amazon.awssdk.http.apache.ApacheHttpClient;
import software.amazon.awssdk.http.nio.netty.NettyNioAsyncHttpClient;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.*;
import software.amazon.awssdk.services.s3.crt.*;
public class ProxyModeRepro {
static final URI ENDPOINT = URI.create("http://example-s3.invalid"); // plaintext
static final AwsCredentialsProvider CREDS =
StaticCredentialsProvider.create(AwsBasicCredentials.create("ak", "sk"));
public static void main(String[] args) throws Exception {
// CRT -> proxy on 8888
S3AsyncClient.crtBuilder()
.endpointOverride(ENDPOINT).region(Region.US_EAST_1).forcePathStyle(true)
.credentialsProvider(CREDS)
.httpConfiguration(S3CrtHttpConfiguration.builder()
.proxyConfiguration(S3CrtProxyConfiguration.builder()
.scheme("http").host("127.0.0.1").port(8888).build())
.build())
.build()
.listObjectsV2(r -> r.bucket("test-bucket")).exceptionally(t -> null).join();
// Netty -> proxy on 8889
S3AsyncClient.builder()
.endpointOverride(ENDPOINT).region(Region.US_EAST_1).forcePathStyle(true)
.credentialsProvider(CREDS)
.httpClientBuilder(NettyNioAsyncHttpClient.builder()
.proxyConfiguration(software.amazon.awssdk.http.nio.netty.ProxyConfiguration.builder()
.scheme("http").host("127.0.0.1").port(8889).build()))
.build()
.listObjectsV2(r -> r.bucket("test-bucket")).exceptionally(t -> null).join();
// Apache (sync) -> proxy on 8890
try {
S3Client.builder()
.endpointOverride(ENDPOINT).region(Region.US_EAST_1).forcePathStyle(true)
.credentialsProvider(CREDS)
.httpClientBuilder(ApacheHttpClient.builder()
.proxyConfiguration(software.amazon.awssdk.http.apache.ProxyConfiguration.builder()
.endpoint(URI.create("http://127.0.0.1:8890")).build()))
.build()
.listObjectsV2(r -> r.bucket("test-bucket"));
} catch (Exception ignored) { }
System.exit(0);
}
}
```
**3. Observed output:**
```
port 8888 FIRST LINE: CONNECT example-s3.invalid:80 HTTP/1.1 <- CRT (unexpected)
port 8889 FIRST LINE: CONNECT example-s3.invalid:80 HTTP/1.1 <- Netty (unexpected)
port 8890 FIRST LINE: GET http://example-s3.invalid/test-bucket?list-type=2 HTTP/1.1 <- Apache (expected)
```
### Possible Solution
**Root cause (CRT path).** `aws-c-http` decides the proxy connection type from whether TLS options were supplied for the main connection ([`proxy.h`](https://github.com/awslabs/aws-c-http/blob/main/include/aws/http/proxy.h)):
```c
AWS_HPCT_HTTP_LEGACY = 0,
/* If tls options are provided (for the main connection) then treat the proxy as a tunneling proxy
If tls options are not provided (for the main connection), then treat the proxy as a forwarding proxy */
```
`HttpProxyOptions` defaults `connectionType` to `Legacy` and `CrtConfigurationUtils.resolveProxy` never calls `setConnectionType(...)`, so this rule always applies. Meanwhile `S3NativeClientConfiguration` creates a `TlsContext` unconditionally, with no regard for the endpoint scheme:
```java
// S3NativeClientConfiguration.java
clientTlsContextOptions = TlsContextOptions.createDefaultClient()...
this.tlsContext = new TlsContext(clientTlsContextOptions);
```
and `S3CrtAsyncHttpClient` passes it to the CRT S3 client unconditionally:
```java
// S3CrtAsyncHttpClient.java
.withTlsContext(s3NativeClientConfiguration.tlsContext())
```
So the main connection always "has TLS options" and `Legacy` always resolves to tunneling — even though the client will not actually use TLS for an `http://` endpoint. In other words, *"the client holds a TLS context"* is being treated as *"this connection uses TLS"*.
Notably, `aws-crt-java`'s own generic `HttpClientConnectionManager` already gates this correctly by scheme:
```java
// HttpClientConnectionManager.java
boolean useTls = HTTPS.equals(uri.getScheme());
...
useTls && tlsContext != null ? tlsContext.getNativeHandle() : 0,
```
**Suggested fix (verified).** Set the connection type explicitly when the endpoint is plaintext, in `S3NativeClientConfiguration`'s constructor right after `resolveProxy` (`endpointOverride` is already assigned at that point):
```java
this.proxyOptions = resolveProxy(builder.httpConfiguration.proxyConfiguration(), tlsContext).orElse(null);
if (this.proxyOptions != null && this.endpointOverride != null
&& "http".equalsIgnoreCase(this.endpointOverride.getScheme())) {
this.proxyOptions.setConnectionType(HttpProxyConnectionType.Forwarding);
}
```
I verified this directly against the CRT `S3Client` (bypassing the SDK wrapper) with the same stub proxy:
| `connectionType` | `tlsContext` passed | First line sent |
|---|---|---|
| `Legacy` (current) | yes | `CONNECT host:80` |
| `Legacy` | **no** | `CONNECT host:80` |
| **`Forwarding`** | **yes** | **`GET http://host:80/...`** |
| `Forwarding` | no | `GET http://host:80/...` |
Two findings worth highlighting:
- **Simply not passing the `TlsContext` does not help** — `aws-c-s3` still tunnels. The connection type has to be set explicitly.
- **`Forwarding` combined with a non-null `TlsContext` works fine** and does not trigger the "configuration error" mentioned in the `AWS_HPCT_HTTP_FORWARD` doc comment (that refers to TLS on the tunnel *destination*). So the fix does not need to touch `tlsContext` handling at all.
This requires no new public API — `HttpProxyConnectionType.Forwarding` and `HttpProxyOptions.setConnectionType()` are already public in `aws-crt-java`.
For the Netty client the cause is different: `AwaitCloseChannelPoolMap` unconditionally uses `Http1TunnelConnectionPool` whenever a proxy matches, and there is no forwarding implementation in `netty-nio-client` at all. That is a larger change and may warrant a separate issue — happy to split it out if preferred.
**Minor note:** in `Forwarding` mode CRT emits `GET http://host:80/path` with the explicit default port, whereas `ApacheHttpClient` and curl emit `GET http://host/path`. Both are valid absolute-form request targets per RFC 9112, but you may want to normalise this.
### Additional Information/Context
Workaround for affected users: switch to the synchronous `S3Client` with `ApacheHttpClient`, which selects the proxy mode from the endpoint scheme correctly.
I'm happy to submit a PR with the fix and tests if the approach looks acceptable.
### AWS Java SDK version used
`2.54.4` (also reproduced on `2.33.0`); `software.amazon.awssdk.crt:aws-crt:0.48.4`
### JDK version used
```
openjdk version "11.0.31" 2026-04-21
OpenJDK Runtime Environment (build 11.0.31+11-post-1ubuntu1-24.04.2-Ubuntu)
```
### Operating System and version
Ubuntu 24.04 (kernel 6.17)
Guida per i contributori
Apri la guida per i contributori
Direzione di ricerca
Inizia con S3NativeClientConfiguration.java e S3CrtAsyncHttpClient.java, riproducendo i casi di proxy HTTP e HTTPS con lo stub_proxy.py fornito. Confronta la gestione dello schema con HttpClientConnectionManager.java, quindi esamina AwaitCloseChannelPoolMap per il percorso proxy di Netty. Il lavoro è completato quando gli endpoint in testo in chiaro usano il forwarding, mentre gli endpoint HTTPS mantengono il comportamento CONNECT per entrambi i client asincroni.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- aws, java
- Ambito
- backend, networking
- Tipo di issue
- Bug
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Stato di attività
- Attiva
- Chiarezza
- Specificata chiaramente
- Idoneità per principianti
- 58/100