awslabs / awslabs/aws-sdk-rust

Endpoint URL with path component causes incorrect request URLs for S3-compatible services

Open
#1,387 0 comments 0 reactions 0 assignees View on GitHub
bug p3
Dominant language
Rust
Stars
3.3k
Forks
290
Avg merge
1d 12h
Merged PRs (30d)
3

Description

### Describe the bug

When using S3-compatible services that expose their S3 API behind a path prefix (e.g., `https://project.supabase.co/storage/v1/s3`), the AWS SDK for Rust does not work, making it incompatible with these services. Using libraries like boto3 works.

This is similar to issues reported in other AWS SDK implementations:
- AWS SDK PHP: [Issue #3037](https://github.com/aws/aws-sdk-php/issues/3037) and [Issue #3099](https://github.com/aws/aws-sdk-php/issues/3099)
- AWS SDK JavaScript: [Issue #4061](https://github.com/aws/aws-sdk-js-v3/issues/4061

### Regression Issue

- [ ] Select this option if this issue appears to be a regression.

### Expected Behavior

When configuring an endpoint URL with a path component:
```rust
let config = aws_sdk_s3::Config::builder()
.endpoint_url("https://project.supabase.co/storage/v1/s3")
.region(Region::new("us-east-1"))
.credentials_provider(credentials)
.build();

let client = aws_sdk_s3::Client::from_conf(config);
```

Accessing bucket `my-bucket` and key `file.parquet` should construct the URL:
```
https://project.supabase.co/storage/v1/s3/my-bucket/file.parquet
```

### Current Behavior

Currently this works with `https://project.supabase.co/storage/v1/s3/` but not `https://project.supabase.co/storage/v1/s3`

### Reproduction Steps

I scrubbed out my credentials, but running this fails
```rust
let credentials = Credentials::new(
access_key_id,
secret_access_key,
None,
None,
"static",
);

let config = Config::builder()
.endpoint_url(endpoint)
.force_path_style(true)
.region(Region::new("us-east-1"))
.credentials_provider(credentials.clone())
.build();

let client = Client::from_conf(config);

match client
.head_object()
.bucket(bucket)
.key(key)
.send()
.await
{
Ok(output) => {
println!("SUCCESS!");
}
Err(e) => {
println!("FAILED!");
}
}
```

and rerunning the same with `let endpoint = format!("{}/", endpoint);` succeeds.

### Possible Solution

_No response_

### Additional Information/Context

_No response_

### Version

```text
; cargo tree | grep aws-
│ │ ├── aws-credential-types v1.2.10
│ │ │ ├── aws-smithy-async v1.2.6
│ │ │ ├── aws-smithy-runtime-api v1.9.2
│ │ │ │ ├── aws-smithy-async v1.2.6 (*)
│ │ │ │ ├── aws-smithy-types v1.3.4
│ │ │ ├── aws-smithy-types v1.3.4 (*)
│ │ │ │ │ ├── aws-lc-rs v1.14.1
│ │ │ │ │ │ ├── aws-lc-sys v0.32.0
│ │ │ │ │ │ ├── aws-lc-rs v1.14.1 (*)
│ │ │ ├── aws-config v1.8.1
│ │ │ │ ├── aws-credential-types v1.2.10 (*)
│ │ │ │ ├── aws-runtime v1.5.16
│ │ │ │ │ ├── aws-credential-types v1.2.10 (*)
│ │ │ │ │ ├── aws-sigv4 v1.3.6
│ │ │ │ │ │ ├── aws-credential-types v1.2.10 (*)
│ │ │ │ │ │ ├── aws-smithy-eventstream v0.60.13
│ │ │ │ │ │ │ ├── aws-smithy-types v1.3.4 (*)
│ │ │ │ │ │ ├── aws-smithy-http v0.62.5
│ │ │ │ │ │ │ ├── aws-smithy-eventstream v0.60.13 (*)
│ │ │ │ │ │ │ ├── aws-smithy-runtime-api v1.9.2 (*)
│ │ │ │ │ │ │ ├── aws-smithy-types v1.3.4 (*)
│ │ │ │ │ │ ├── aws-smithy-runtime-api v1.9.2 (*)
│ │ │ │ │ │ ├── aws-smithy-types v1.3.4 (*)
│ │ │ │ │ ├── aws-smithy-async v1.2.6 (*)
│ │ │ │ │ ├── aws-smithy-eventstream v0.60.13 (*)
│ │ │ │ │ ├── aws-smithy-http v0.62.5 (*)
│ │ │ │ │ ├── aws-smithy-runtime v1.9.4
│ │ │ │ │ │ ├── aws-smithy-async v1.2.6 (*)
│ │ │ │ │ │ ├── aws-smithy-http v0.62.5 (*)
│ │ │ │ │ │ ├── aws-smithy-http-client v1.1.4
│ │ │ │ │ │ │ ├── aws-smithy-async v1.2.6 (*)
│ │ │ │ │ │ │ ├── aws-smithy-runtime-api v1.9.2 (*)
│ │ │ │ │ │ │ ├── aws-smithy-types v1.3.4 (*)
│ │ │ │ │ │ ├── aws-smithy-observability v0.1.4
│ │ │ │ │ │ │ └── aws-smithy-runtime-api v1.9.2 (*)
│ │ │ │ │ │ ├── aws-smithy-runtime-api v1.9.2 (*)
│ │ │ │ │ │ ├── aws-smithy-types v1.3.4 (*)
│ │ │ │ │ ├── aws-smithy-runtime-api v1.9.2 (*)
│ │ │ │ │ ├── aws-smithy-types v1.3.4 (*)
│ │ │ │ │ ├── aws-types v1.3.10
│ │ │ │ │ │ ├── aws-credential-types v1.2.10 (*)
│ │ │ │ │ │ ├── aws-smithy-async v1.2.6 (*)
│ │ │ │ │ │ ├── aws-smithy-runtime-api v1.9.2 (*)
│ │ │ │ │ │ ├── aws-smithy-types v1.3.4 (*)
│ │ │ │ ├── aws-sdk-sso v1.74.0
│ │ │ │ │ ├── aws-credential-types v1.2.10 (*)
│ │ │ │ │ ├── aws-runtime v1.5.16 (*)
│ │ │ │ │ ├── aws-smithy-async v1.2.6 (*)
│ │ │ │ │ ├── aws-smithy-http v0.62.5 (*)
│ │ │ │ │ ├── aws-smithy-json v0.61.7
│ │ │ │ │ │ └── aws-smithy-types v1.3.4 (*)
│ │ │ │ │ ├── aws-smithy-runtime v1.9.4 (*)
│ │ │ │ │ ├── aws-smithy-runtime-api v1.9.2 (*)
│ │ │ │ │ ├── aws-smithy-types v1.3.4 (*)
│ │ │ │ │ ├── aws-types v1.3.10 (*)
│ │ │ │ ├── aws-sdk-ssooidc v1.75.0
│ │ │ │ │ ├── aws-credential-types v1.2.10 (*)
│ │ │ │ │ ├── aws-runtime v1.5.16 (*)
│ │ │ │ │ ├── aws-smithy-async v1.2.6 (*)
│ │ │ │ │ ├── aws-smithy-http v0.62.5 (*)
│ │ │ │ │ ├── aws-smithy-json v0.61.7 (*)
│ │ │ │ │ ├── aws-smithy-runtime v1.9.4 (*)
│ │ │ │ │ ├── aws-smithy-runtime-api v1.9.2 (*)
│ │ │ │ │ ├── aws-smithy-types v1.3.4 (*)
│ │ │ │ │ ├── aws-types v1.3.10 (*)
│ │ │ │ ├── aws-sdk-sts v1.76.0
│ │ │ │ │ ├── aws-credential-types v1.2.10 (*)
│ │ │ │ │ ├── aws-runtime v1.5.16 (*)
│ │ │ │ │ ├── aws-smithy-async v1.2.6 (*)
│ │ │ │ │ ├── aws-smithy-http v0.62.5 (*)
│ │ │ │ │ ├── aws-smithy-json v0.61.7 (*)
│ │ │ │ │ ├── aws-smithy-query v0.60.7
│ │ │ │ │ │ ├── aws-smithy-types v1.3.4 (*)
│ │ │ │ │ ├── aws-smithy-runtime v1.9.4 (*)
│ │ │ │ │ ├── aws-smithy-runtime-api v1.9.2 (*)
│ │ │ │ │ ├── aws-smithy-types v1.3.4 (*)
│ │ │ │ │ ├── aws-smithy-xml v0.60.12
│ │ │ │ │ ├── aws-types v1.3.10 (*)
│ │ │ │ ├── aws-smithy-async v1.2.6 (*)
│ │ │ │ ├── aws-smithy-http v0.62.5 (*)
│ │ │ │ ├── aws-smithy-json v0.61.7 (*)
│ │ │ │ ├── aws-smithy-runtime v1.9.4 (*)
│ │ │ │ ├── aws-smithy-runtime-api v1.9.2 (*)
│ │ │ │ ├── aws-smithy-types v1.3.4 (*)
│ │ │ │ ├── aws-types v1.3.10 (*)
│ │ │ ├── aws-credential-types v1.2.10 (*)
│ │ │ ├── aws-sdk-s3 v1.115.0
│ │ │ │ ├── aws-credential-types v1.2.10 (*)
│ │ │ │ ├── aws-runtime v1.5.16 (*)
│ │ │ │ ├── aws-sigv4 v1.3.6 (*)
│ │ │ │ ├── aws-smithy-async v1.2.6 (*)
│ │ │ │ ├── aws-smithy-checksums v0.63.11
│ │ │ │ │ ├── aws-smithy-http v0.62.5 (*)
│ │ │ │ │ ├── aws-smithy-types v1.3.4 (*)
│ │ │ │ ├── aws-smithy-eventstream v0.60.13 (*)
│ │ │ │ ├── aws-smithy-http v0.62.5 (*)
│ │ │ │ ├── aws-smithy-json v0.61.7 (*)
│ │ │ │ ├── aws-smithy-runtime v1.9.4 (*)
│ │ │ │ ├── aws-smithy-runtime-api v1.9.2 (*)
│ │ │ │ ├── aws-smithy-types v1.3.4 (*)
│ │ │ │ ├── aws-smithy-xml v0.60.12 (*)
│ │ │ │ ├── aws-types v1.3.10 (*)
│ │ │ ├── aws-smithy-http-client v1.1.4 (*)
│ │ │ ├── aws-smithy-runtime-api v1.9.2 (*)
```

### Environment details (OS name and version, etc.)

MacOS version 15.6

### Logs

_No response_

Contributor guide

Open the contributing guide

Research direction

Start at Config::builder endpoint_url handling and trace how Client::from_conf builds the head_object request when force_path_style is enabled. Add regression coverage for an endpoint with and without a trailing slash, and verify that the bucket and key are appended to the expected path without changing existing behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, rust
Domain
api, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.