S3 async clients (CRT and Netty) always use CONNECT tunneling with a proxy, even for plaintext http:// endpoints
- Dominant language
- Java
- Stars
- 2.6k
- Forks
- 1k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 51
Description
### Describe the bug
When an HTTP proxy is configured, the asynchronous S3 clients (`S3AsyncClient.crtBuilder()` and `S3AsyncClient.builder()`) always establish the proxy connection with a `CONNECT` tunnel, even when `endpointOverride` uses the plaintext `http://` scheme. The synchronous `S3Client` backed by `ApacheHttpClient` correctly uses forwarding mode for the same configuration.
Many corporate proxies deliberately disallow the `CONNECT` method (tunnels defeat traffic inspection and can be used to bypass egress controls) and only support forwarding. Against such a proxy the async S3 clients cannot be used at all, while the sync client works.
Note that the emitted request is self-contradictory: the client sends `CONNECT host:80` — it knows the target is plaintext (hence port 80) yet still negotiates a tunnel as if TLS were in use.
### Regression Issue
- [ ] Select this option if this issue appears to be a regression.
Not a regression — reproduced identically on 2.33.0 and 2.54.4.
### Expected Behavior
With `endpointOverride = http://` and a proxy configured, the client should use forwarding mode and send an absolute-form request, matching both `curl -x http:// http:///` and the SDK's own `ApacheHttpClient`:
```
GET http:///?list-type=2 HTTP/1.1
```
### Current Behavior
The client sends:
```
CONNECT :80 HTTP/1.1
```
Against a proxy that rejects `CONNECT`, the request fails with:
```
software.amazon.awssdk.core.exception.SdkClientException:
Failed to send the request: Proxy-based connection establishment failed because the CONNECT call failed
```
**Behavior matrix observed (SDK 2.54.4, aws-crt 0.48.4)** — identical configuration, only the HTTP client differs:
| HTTP client | `http://` endpoint | `https://` endpoint |
|---|---|---|
| `ApacheHttpClient` (sync `S3Client`) | `GET http://host/...` :white_check_mark: | `CONNECT host:443` :white_check_mark: |
| CRT (`S3AsyncClient.crtBuilder()`) | `CONNECT host:80` :x: | `CONNECT host:443` :white_check_mark: |
| Netty (`S3AsyncClient.builder()`) | `CONNECT host:80` :x: | `CONNECT host:443` :white_check_mark: |
The same CRT client is also inconsistent with itself depending on whether a proxy is configured:
| Scenario | endpoint | First bytes sent |
|---|---|---|
| CRT, **no proxy** | `http://` | `GET /?list-type=2 HTTP/1.1` (plaintext) :white_check_mark: |
| CRT, **no proxy** | `https://` | TLS ClientHello :white_check_mark: |
| CRT, **with proxy** | `http://` | `CONNECT host:80` :x: |
So the CRT client honours the endpoint scheme when connecting directly, but stops honouring it once a proxy is involved.
### Reproduction Steps
**1. Start a minimal proxy stub** that prints the first line it receives and closes the connection:
```python
# stub_proxy.py
import socket, sys, threading
port = int(sys.argv[1])
s = socket.socket(); s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind(("127.0.0.1", port)); s.listen(8)
def handle(c):
data = c.recv(4096)
line = data.split(b"\r\n")[0].decode("latin1") if data else "(empty)"
if data[:1] == b"\x16": line = ""
print(f"port {port} FIRST LINE: {line}", flush=True)
c.close()
while True:
c, _ = s.accept(); threading.Thread(target=handle, args=(c,), daemon=True).start()
```
```bash
python3 stub_proxy.py 8888 &
python3 stub_proxy.py 8889 &
python3 stub_proxy.py 8890 &
```
**2. Run each client against a plaintext endpoint:**
```java
import java.net.URI;
import software.amazon.awssdk.auth.credentials.*;
import software.amazon.awssdk.http.apache.ApacheHttpClient;
import software.amazon.awssdk.http.nio.netty.NettyNioAsyncHttpClient;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.*;
import software.amazon.awssdk.services.s3.crt.*;
public class ProxyModeRepro {
static final URI ENDPOINT = URI.create("http://example-s3.invalid"); // plaintext
static final AwsCredentialsProvider CREDS =
StaticCredentialsProvider.create(AwsBasicCredentials.create("ak", "sk"));
public static void main(String[] args) throws Exception {
// CRT -> proxy on 8888
S3AsyncClient.crtBuilder()
.endpointOverride(ENDPOINT).region(Region.US_EAST_1).forcePathStyle(true)
.credentialsProvider(CREDS)
.httpConfiguration(S3CrtHttpConfiguration.builder()
.proxyConfiguration(S3CrtProxyConfiguration.builder()
.scheme("http").host("127.0.0.1").port(8888).build())
.build())
.build()
.listObjectsV2(r -> r.bucket("test-bucket")).exceptionally(t -> null).join();
// Netty -> proxy on 8889
S3AsyncClient.builder()
.endpointOverride(ENDPOINT).region(Region.US_EAST_1).forcePathStyle(true)
.credentialsProvider(CREDS)
.httpClientBuilder(NettyNioAsyncHttpClient.builder()
.proxyConfiguration(software.amazon.awssdk.http.nio.netty.ProxyConfiguration.builder()
.scheme("http").host("127.0.0.1").port(8889).build()))
.build()
.listObjectsV2(r -> r.bucket("test-bucket")).exceptionally(t -> null).join();
// Apache (sync) -> proxy on 8890
try {
S3Client.builder()
.endpointOverride(ENDPOINT).region(Region.US_EAST_1).forcePathStyle(true)
.credentialsProvider(CREDS)
.httpClientBuilder(ApacheHttpClient.builder()
.proxyConfiguration(software.amazon.awssdk.http.apache.ProxyConfiguration.builder()
.endpoint(URI.create("http://127.0.0.1:8890")).build()))
.build()
.listObjectsV2(r -> r.bucket("test-bucket"));
} catch (Exception ignored) { }
System.exit(0);
}
}
```
**3. Observed output:**
```
port 8888 FIRST LINE: CONNECT example-s3.invalid:80 HTTP/1.1 <- CRT (unexpected)
port 8889 FIRST LINE: CONNECT example-s3.invalid:80 HTTP/1.1 <- Netty (unexpected)
port 8890 FIRST LINE: GET http://example-s3.invalid/test-bucket?list-type=2 HTTP/1.1 <- Apache (expected)
```
### Possible Solution
**Root cause (CRT path).** `aws-c-http` decides the proxy connection type from whether TLS options were supplied for the main connection ([`proxy.h`](https://github.com/awslabs/aws-c-http/blob/main/include/aws/http/proxy.h)):
```c
AWS_HPCT_HTTP_LEGACY = 0,
/* If tls options are provided (for the main connection) then treat the proxy as a tunneling proxy
If tls options are not provided (for the main connection), then treat the proxy as a forwarding proxy */
```
`HttpProxyOptions` defaults `connectionType` to `Legacy` and `CrtConfigurationUtils.resolveProxy` never calls `setConnectionType(...)`, so this rule always applies. Meanwhile `S3NativeClientConfiguration` creates a `TlsContext` unconditionally, with no regard for the endpoint scheme:
```java
// S3NativeClientConfiguration.java
clientTlsContextOptions = TlsContextOptions.createDefaultClient()...
this.tlsContext = new TlsContext(clientTlsContextOptions);
```
and `S3CrtAsyncHttpClient` passes it to the CRT S3 client unconditionally:
```java
// S3CrtAsyncHttpClient.java
.withTlsContext(s3NativeClientConfiguration.tlsContext())
```
So the main connection always "has TLS options" and `Legacy` always resolves to tunneling — even though the client will not actually use TLS for an `http://` endpoint. In other words, *"the client holds a TLS context"* is being treated as *"this connection uses TLS"*.
Notably, `aws-crt-java`'s own generic `HttpClientConnectionManager` already gates this correctly by scheme:
```java
// HttpClientConnectionManager.java
boolean useTls = HTTPS.equals(uri.getScheme());
...
useTls && tlsContext != null ? tlsContext.getNativeHandle() : 0,
```
**Suggested fix (verified).** Set the connection type explicitly when the endpoint is plaintext, in `S3NativeClientConfiguration`'s constructor right after `resolveProxy` (`endpointOverride` is already assigned at that point):
```java
this.proxyOptions = resolveProxy(builder.httpConfiguration.proxyConfiguration(), tlsContext).orElse(null);
if (this.proxyOptions != null && this.endpointOverride != null
&& "http".equalsIgnoreCase(this.endpointOverride.getScheme())) {
this.proxyOptions.setConnectionType(HttpProxyConnectionType.Forwarding);
}
```
I verified this directly against the CRT `S3Client` (bypassing the SDK wrapper) with the same stub proxy:
| `connectionType` | `tlsContext` passed | First line sent |
|---|---|---|
| `Legacy` (current) | yes | `CONNECT host:80` |
| `Legacy` | **no** | `CONNECT host:80` |
| **`Forwarding`** | **yes** | **`GET http://host:80/...`** |
| `Forwarding` | no | `GET http://host:80/...` |
Two findings worth highlighting:
- **Simply not passing the `TlsContext` does not help** — `aws-c-s3` still tunnels. The connection type has to be set explicitly.
- **`Forwarding` combined with a non-null `TlsContext` works fine** and does not trigger the "configuration error" mentioned in the `AWS_HPCT_HTTP_FORWARD` doc comment (that refers to TLS on the tunnel *destination*). So the fix does not need to touch `tlsContext` handling at all.
This requires no new public API — `HttpProxyConnectionType.Forwarding` and `HttpProxyOptions.setConnectionType()` are already public in `aws-crt-java`.
For the Netty client the cause is different: `AwaitCloseChannelPoolMap` unconditionally uses `Http1TunnelConnectionPool` whenever a proxy matches, and there is no forwarding implementation in `netty-nio-client` at all. That is a larger change and may warrant a separate issue — happy to split it out if preferred.
**Minor note:** in `Forwarding` mode CRT emits `GET http://host:80/path` with the explicit default port, whereas `ApacheHttpClient` and curl emit `GET http://host/path`. Both are valid absolute-form request targets per RFC 9112, but you may want to normalise this.
### Additional Information/Context
Workaround for affected users: switch to the synchronous `S3Client` with `ApacheHttpClient`, which selects the proxy mode from the endpoint scheme correctly.
I'm happy to submit a PR with the fix and tests if the approach looks acceptable.
### AWS Java SDK version used
`2.54.4` (also reproduced on `2.33.0`); `software.amazon.awssdk.crt:aws-crt:0.48.4`
### JDK version used
```
openjdk version "11.0.31" 2026-04-21
OpenJDK Runtime Environment (build 11.0.31+11-post-1ubuntu1-24.04.2-Ubuntu)
```
### Operating System and version
Ubuntu 24.04 (kernel 6.17)
Contributor guide
Research direction
Start with S3NativeClientConfiguration.java and S3CrtAsyncHttpClient.java, reproducing the HTTP and HTTPS proxy cases with the provided stub_proxy.py. Compare the scheme handling with HttpClientConnectionManager.java, then inspect AwaitCloseChannelPoolMap for Netty's proxy path. Done means plaintext endpoints use forwarding while HTTPS endpoints retain CONNECT behavior for both async clients.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java
- Domain
- backend, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 58/100