apache / apache/arrow-rs-object-store
AmazonS3Builder::with_url accepts URLs with paths but ignores those paths
- Dominant language
- Rust
- Stars
- 322
- Forks
- 212
- Avg merge
- 5d 2h
- Merged PRs (30d)
- 10
Description
**Describe the bug**
The docs for `AmazonS3Builder::with_url` show the following example:
```
use object_store::aws::AmazonS3Builder;
let s3 = AmazonS3Builder::from_env()
.with_url("s3://bucket/path")
.build();
```
But this produces an S3 store that ignores the `/path` part.
output of `dbg!(s3)`
```
[src/main.rs:22:5] s3 = AmazonS3 {
client: S3Client {
config: S3Config {
region: "us-east-1",
endpoint: None,
bucket: "bucket",
bucket_endpoint: "https://s3.us-east-1.amazonaws.com/bucket",
credentials: TokenCredentialProvider {
inner: InstanceCredentialProvider {
imdsv1_fallback: false,
metadata_endpoint: "http://169.254.169.254",
},
client: HttpClient(
Client {
accepts: Accepts,
proxies: [
Matcher,
],
referer: true,
default_headers: {
"accept": "*/*",
"user-agent": "object_store/0.12.2",
},
reqwest::config::RequestTimeout: 30s,
},
),
retry: RetryConfig {
backoff: BackoffConfig {
init_backoff: 100ms,
max_backoff: 15s,
base: 2.0,
},
max_retries: 10,
retry_timeout: 180s,
},
cache: TokenCache {
cache: Mutex {
data: None,
},
min_ttl: 300s,
fetch_backoff: 100ms,
},
},
session_provider: None,
retry_config: RetryConfig {
backoff: BackoffConfig {
init_backoff: 100ms,
max_backoff: 15s,
base: 2.0,
},
max_retries: 10,
retry_timeout: 180s,
},
client_options: ClientOptions {
user_agent: None,
root_certificates: [],
content_type_map: {},
default_content_type: None,
default_headers: None,
proxy_url: None,
proxy_ca_certificate: None,
proxy_excludes: None,
allow_http: Parsed(
false,
),
allow_insecure: Parsed(
false,
),
timeout: Some(
Parsed(
30s,
),
),
connect_timeout: Some(
Parsed(
5s,
),
),
pool_idle_timeout: None,
pool_max_idle_per_host: None,
http2_keep_alive_interval: None,
http2_keep_alive_timeout: None,
http2_keep_alive_while_idle: Parsed(
false,
),
http2_max_frame_size: None,
http1_only: Parsed(
true,
),
http2_only: Parsed(
false,
),
randomize_addresses: Parsed(
true,
),
},
sign_payload: true,
skip_signature: false,
disable_tagging: false,
checksum: None,
copy_if_not_exists: None,
conditional_put: ETagMatch,
request_payer: false,
encryption_headers: S3EncryptionHeaders(
{},
),
},
client: HttpClient(
Client {
accepts: Accepts,
proxies: [
Matcher,
],
referer: true,
default_headers: {
"accept": "*/*",
"user-agent": "object_store/0.12.2",
},
reqwest::config::RequestTimeout: 30s,
},
),
},
}
```
**To Reproduce**
```
use object_store::aws::AmazonS3Builder;
let s3 = AmazonS3Builder::from_env()
.with_url("s3://bucket/path")
.build()?;
dbg!(s3);
s3.get(&"foo".into()); // observe this tries to fetch s3://bucket/foo not s3://bucket/path/foo
```
**Expected behavior**
I expected to this make build() return the equivalent of a PrefixStore with the given path. Alternatively, it could return an error if given a URL with a path. I'd be happy to make a PR for one of these options if those sound right to you.
Contributor guide
Research direction
Start with AmazonS3Builder::with_url and the s3://bucket/path reproduction; inspect how the URL is converted into the S3 store. Confirm the chosen behavior for the path—prefixing requests or rejecting the URL—and verify that s3.get(&"foo".into()) no longer targets s3://bucket/foo.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, rust
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 50/100