aws / aws/aws-sdk-java-v2

S3 async clients (CRT and Netty) always use CONNECT tunneling with a proxy, even for plaintext http:// endpoints

Offen
#7,320 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Java
Sterne
2.6k
Forks
1k
Ø Merge
2 T. 9 Std.
Gemergte PRs (30 T.)
51

Beschreibung

### Describe the bug

When an HTTP proxy is configured, the asynchronous S3 clients (`S3AsyncClient.crtBuilder()` and `S3AsyncClient.builder()`) always establish the proxy connection with a `CONNECT` tunnel, even when `endpointOverride` uses the plaintext `http://` scheme. The synchronous `S3Client` backed by `ApacheHttpClient` correctly uses forwarding mode for the same configuration.

Many corporate proxies deliberately disallow the `CONNECT` method (tunnels defeat traffic inspection and can be used to bypass egress controls) and only support forwarding. Against such a proxy the async S3 clients cannot be used at all, while the sync client works.

Note that the emitted request is self-contradictory: the client sends `CONNECT host:80` — it knows the target is plaintext (hence port 80) yet still negotiates a tunnel as if TLS were in use.

### Regression Issue

- [ ] Select this option if this issue appears to be a regression.

Not a regression — reproduced identically on 2.33.0 and 2.54.4.

### Expected Behavior

With `endpointOverride = http://` and a proxy configured, the client should use forwarding mode and send an absolute-form request, matching both `curl -x http:// http:///` and the SDK's own `ApacheHttpClient`:

```
GET http:///?list-type=2 HTTP/1.1
```

### Current Behavior

The client sends:

```
CONNECT :80 HTTP/1.1
```

Against a proxy that rejects `CONNECT`, the request fails with:

```
software.amazon.awssdk.core.exception.SdkClientException:
Failed to send the request: Proxy-based connection establishment failed because the CONNECT call failed
```

**Behavior matrix observed (SDK 2.54.4, aws-crt 0.48.4)** — identical configuration, only the HTTP client differs:

| HTTP client | `http://` endpoint | `https://` endpoint |
|---|---|---|
| `ApacheHttpClient` (sync `S3Client`) | `GET http://host/...` :white_check_mark: | `CONNECT host:443` :white_check_mark: |
| CRT (`S3AsyncClient.crtBuilder()`) | `CONNECT host:80` :x: | `CONNECT host:443` :white_check_mark: |
| Netty (`S3AsyncClient.builder()`) | `CONNECT host:80` :x: | `CONNECT host:443` :white_check_mark: |

The same CRT client is also inconsistent with itself depending on whether a proxy is configured:

| Scenario | endpoint | First bytes sent |
|---|---|---|
| CRT, **no proxy** | `http://` | `GET /?list-type=2 HTTP/1.1` (plaintext) :white_check_mark: |
| CRT, **no proxy** | `https://` | TLS ClientHello :white_check_mark: |
| CRT, **with proxy** | `http://` | `CONNECT host:80` :x: |

So the CRT client honours the endpoint scheme when connecting directly, but stops honouring it once a proxy is involved.

### Reproduction Steps

**1. Start a minimal proxy stub** that prints the first line it receives and closes the connection:

```python
# stub_proxy.py
import socket, sys, threading
port = int(sys.argv[1])
s = socket.socket(); s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind(("127.0.0.1", port)); s.listen(8)
def handle(c):
data = c.recv(4096)
line = data.split(b"\r\n")[0].decode("latin1") if data else "(empty)"
if data[:1] == b"\x16": line = ""
print(f"port {port} FIRST LINE: {line}", flush=True)
c.close()
while True:
c, _ = s.accept(); threading.Thread(target=handle, args=(c,), daemon=True).start()
```

```bash
python3 stub_proxy.py 8888 &
python3 stub_proxy.py 8889 &
python3 stub_proxy.py 8890 &
```

**2. Run each client against a plaintext endpoint:**

```java
import java.net.URI;
import software.amazon.awssdk.auth.credentials.*;
import software.amazon.awssdk.http.apache.ApacheHttpClient;
import software.amazon.awssdk.http.nio.netty.NettyNioAsyncHttpClient;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.*;
import software.amazon.awssdk.services.s3.crt.*;

public class ProxyModeRepro {
static final URI ENDPOINT = URI.create("http://example-s3.invalid"); // plaintext
static final AwsCredentialsProvider CREDS =
StaticCredentialsProvider.create(AwsBasicCredentials.create("ak", "sk"));

public static void main(String[] args) throws Exception {
// CRT -> proxy on 8888
S3AsyncClient.crtBuilder()
.endpointOverride(ENDPOINT).region(Region.US_EAST_1).forcePathStyle(true)
.credentialsProvider(CREDS)
.httpConfiguration(S3CrtHttpConfiguration.builder()
.proxyConfiguration(S3CrtProxyConfiguration.builder()
.scheme("http").host("127.0.0.1").port(8888).build())
.build())
.build()
.listObjectsV2(r -> r.bucket("test-bucket")).exceptionally(t -> null).join();

// Netty -> proxy on 8889
S3AsyncClient.builder()
.endpointOverride(ENDPOINT).region(Region.US_EAST_1).forcePathStyle(true)
.credentialsProvider(CREDS)
.httpClientBuilder(NettyNioAsyncHttpClient.builder()
.proxyConfiguration(software.amazon.awssdk.http.nio.netty.ProxyConfiguration.builder()
.scheme("http").host("127.0.0.1").port(8889).build()))
.build()
.listObjectsV2(r -> r.bucket("test-bucket")).exceptionally(t -> null).join();

// Apache (sync) -> proxy on 8890
try {
S3Client.builder()
.endpointOverride(ENDPOINT).region(Region.US_EAST_1).forcePathStyle(true)
.credentialsProvider(CREDS)
.httpClientBuilder(ApacheHttpClient.builder()
.proxyConfiguration(software.amazon.awssdk.http.apache.ProxyConfiguration.builder()
.endpoint(URI.create("http://127.0.0.1:8890")).build()))
.build()
.listObjectsV2(r -> r.bucket("test-bucket"));
} catch (Exception ignored) { }
System.exit(0);
}
}
```

**3. Observed output:**

```
port 8888 FIRST LINE: CONNECT example-s3.invalid:80 HTTP/1.1 <- CRT (unexpected)
port 8889 FIRST LINE: CONNECT example-s3.invalid:80 HTTP/1.1 <- Netty (unexpected)
port 8890 FIRST LINE: GET http://example-s3.invalid/test-bucket?list-type=2 HTTP/1.1 <- Apache (expected)
```

### Possible Solution

**Root cause (CRT path).** `aws-c-http` decides the proxy connection type from whether TLS options were supplied for the main connection ([`proxy.h`](https://github.com/awslabs/aws-c-http/blob/main/include/aws/http/proxy.h)):

```c
AWS_HPCT_HTTP_LEGACY = 0,
/* If tls options are provided (for the main connection) then treat the proxy as a tunneling proxy
If tls options are not provided (for the main connection), then treat the proxy as a forwarding proxy */
```

`HttpProxyOptions` defaults `connectionType` to `Legacy` and `CrtConfigurationUtils.resolveProxy` never calls `setConnectionType(...)`, so this rule always applies. Meanwhile `S3NativeClientConfiguration` creates a `TlsContext` unconditionally, with no regard for the endpoint scheme:

```java
// S3NativeClientConfiguration.java
clientTlsContextOptions = TlsContextOptions.createDefaultClient()...
this.tlsContext = new TlsContext(clientTlsContextOptions);
```

and `S3CrtAsyncHttpClient` passes it to the CRT S3 client unconditionally:

```java
// S3CrtAsyncHttpClient.java
.withTlsContext(s3NativeClientConfiguration.tlsContext())
```

So the main connection always "has TLS options" and `Legacy` always resolves to tunneling — even though the client will not actually use TLS for an `http://` endpoint. In other words, *"the client holds a TLS context"* is being treated as *"this connection uses TLS"*.

Notably, `aws-crt-java`'s own generic `HttpClientConnectionManager` already gates this correctly by scheme:

```java
// HttpClientConnectionManager.java
boolean useTls = HTTPS.equals(uri.getScheme());
...
useTls && tlsContext != null ? tlsContext.getNativeHandle() : 0,
```

**Suggested fix (verified).** Set the connection type explicitly when the endpoint is plaintext, in `S3NativeClientConfiguration`'s constructor right after `resolveProxy` (`endpointOverride` is already assigned at that point):

```java
this.proxyOptions = resolveProxy(builder.httpConfiguration.proxyConfiguration(), tlsContext).orElse(null);
if (this.proxyOptions != null && this.endpointOverride != null
&& "http".equalsIgnoreCase(this.endpointOverride.getScheme())) {
this.proxyOptions.setConnectionType(HttpProxyConnectionType.Forwarding);
}
```

I verified this directly against the CRT `S3Client` (bypassing the SDK wrapper) with the same stub proxy:

| `connectionType` | `tlsContext` passed | First line sent |
|---|---|---|
| `Legacy` (current) | yes | `CONNECT host:80` |
| `Legacy` | **no** | `CONNECT host:80` |
| **`Forwarding`** | **yes** | **`GET http://host:80/...`** |
| `Forwarding` | no | `GET http://host:80/...` |

Two findings worth highlighting:

- **Simply not passing the `TlsContext` does not help** — `aws-c-s3` still tunnels. The connection type has to be set explicitly.
- **`Forwarding` combined with a non-null `TlsContext` works fine** and does not trigger the "configuration error" mentioned in the `AWS_HPCT_HTTP_FORWARD` doc comment (that refers to TLS on the tunnel *destination*). So the fix does not need to touch `tlsContext` handling at all.

This requires no new public API — `HttpProxyConnectionType.Forwarding` and `HttpProxyOptions.setConnectionType()` are already public in `aws-crt-java`.

For the Netty client the cause is different: `AwaitCloseChannelPoolMap` unconditionally uses `Http1TunnelConnectionPool` whenever a proxy matches, and there is no forwarding implementation in `netty-nio-client` at all. That is a larger change and may warrant a separate issue — happy to split it out if preferred.

**Minor note:** in `Forwarding` mode CRT emits `GET http://host:80/path` with the explicit default port, whereas `ApacheHttpClient` and curl emit `GET http://host/path`. Both are valid absolute-form request targets per RFC 9112, but you may want to normalise this.

### Additional Information/Context

Workaround for affected users: switch to the synchronous `S3Client` with `ApacheHttpClient`, which selects the proxy mode from the endpoint scheme correctly.

I'm happy to submit a PR with the fix and tests if the approach looks acceptable.

### AWS Java SDK version used

`2.54.4` (also reproduced on `2.33.0`); `software.amazon.awssdk.crt:aws-crt:0.48.4`

### JDK version used

```
openjdk version "11.0.31" 2026-04-21
OpenJDK Runtime Environment (build 11.0.31+11-post-1ubuntu1-24.04.2-Ubuntu)
```

### Operating System and version

Ubuntu 24.04 (kernel 6.17)

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Beginne mit S3NativeClientConfiguration.java und S3CrtAsyncHttpClient.java, indem du die HTTP- und HTTPS-Proxy-Fälle mit dem bereitgestellten stub_proxy.py nachstellst. Vergleiche die Behandlung des Schemas mit HttpClientConnectionManager.java und untersuche anschließend AwaitCloseChannelPoolMap für den Proxy-Pfad von Netty. Als abgeschlossen gilt die Aufgabe, wenn Klartext-Endpunkte Forwarding verwenden, während HTTPS-Endpunkte für beide asynchronen Clients weiterhin das CONNECT-Verhalten beibehalten.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
aws, java
Bereich
backend, networking
Issue-Typ
Bug
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Aktiv
Klarheit
Klar beschrieben
Anfängerfreundlichkeit
58/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.