apache / apache/arrow-rs-object-store

AmazonS3Builder::with_url accepts URLs with paths but ignores those paths

Open
#409 8 comments 0 reactions 0 assignees View on GitHub
documentation good first issue
Dominant language
Rust
Stars
322
Forks
212
Avg merge
5d 2h
Merged PRs (30d)
10

Description

**Describe the bug**
The docs for `AmazonS3Builder::with_url` show the following example:

```
use object_store::aws::AmazonS3Builder;

let s3 = AmazonS3Builder::from_env()
.with_url("s3://bucket/path")
.build();
```

But this produces an S3 store that ignores the `/path` part.

output of `dbg!(s3)`

```
[src/main.rs:22:5] s3 = AmazonS3 {
client: S3Client {
config: S3Config {
region: "us-east-1",
endpoint: None,
bucket: "bucket",
bucket_endpoint: "https://s3.us-east-1.amazonaws.com/bucket",
credentials: TokenCredentialProvider {
inner: InstanceCredentialProvider {
imdsv1_fallback: false,
metadata_endpoint: "http://169.254.169.254",
},
client: HttpClient(
Client {
accepts: Accepts,
proxies: [
Matcher,
],
referer: true,
default_headers: {
"accept": "*/*",
"user-agent": "object_store/0.12.2",
},
reqwest::config::RequestTimeout: 30s,
},
),
retry: RetryConfig {
backoff: BackoffConfig {
init_backoff: 100ms,
max_backoff: 15s,
base: 2.0,
},
max_retries: 10,
retry_timeout: 180s,
},
cache: TokenCache {
cache: Mutex {
data: None,
},
min_ttl: 300s,
fetch_backoff: 100ms,
},
},
session_provider: None,
retry_config: RetryConfig {
backoff: BackoffConfig {
init_backoff: 100ms,
max_backoff: 15s,
base: 2.0,
},
max_retries: 10,
retry_timeout: 180s,
},
client_options: ClientOptions {
user_agent: None,
root_certificates: [],
content_type_map: {},
default_content_type: None,
default_headers: None,
proxy_url: None,
proxy_ca_certificate: None,
proxy_excludes: None,
allow_http: Parsed(
false,
),
allow_insecure: Parsed(
false,
),
timeout: Some(
Parsed(
30s,
),
),
connect_timeout: Some(
Parsed(
5s,
),
),
pool_idle_timeout: None,
pool_max_idle_per_host: None,
http2_keep_alive_interval: None,
http2_keep_alive_timeout: None,
http2_keep_alive_while_idle: Parsed(
false,
),
http2_max_frame_size: None,
http1_only: Parsed(
true,
),
http2_only: Parsed(
false,
),
randomize_addresses: Parsed(
true,
),
},
sign_payload: true,
skip_signature: false,
disable_tagging: false,
checksum: None,
copy_if_not_exists: None,
conditional_put: ETagMatch,
request_payer: false,
encryption_headers: S3EncryptionHeaders(
{},
),
},
client: HttpClient(
Client {
accepts: Accepts,
proxies: [
Matcher,
],
referer: true,
default_headers: {
"accept": "*/*",
"user-agent": "object_store/0.12.2",
},
reqwest::config::RequestTimeout: 30s,
},
),
},
}
```

**To Reproduce**
```
use object_store::aws::AmazonS3Builder;

let s3 = AmazonS3Builder::from_env()
.with_url("s3://bucket/path")
.build()?;
dbg!(s3);
s3.get(&"foo".into()); // observe this tries to fetch s3://bucket/foo not s3://bucket/path/foo
```
**Expected behavior**
I expected to this make build() return the equivalent of a PrefixStore with the given path. Alternatively, it could return an error if given a URL with a path. I'd be happy to make a PR for one of these options if those sound right to you.

Contributor guide

Open the contributing guide

Research direction

Start with AmazonS3Builder::with_url and the s3://bucket/path reproduction; inspect how the URL is converted into the S3 store. Confirm the chosen behavior for the path—prefixing requests or rejecting the URL—and verify that s3.get(&"foo".into()) no longer targets s3://bucket/foo.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, rust
Domain
cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.