apache / apache/arrow-rs-object-store

Automatically detect S3 regions / redirect correctly

Open
#402 7 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
322
Forks
212
Avg merge
5d 2h
Merged PRs (30d)
10

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
- related to https://github.com/apache/datafusion/issues/16306

I would like to be able to retrieve data stored in S3 without having to explicitly specify a region (see https://github.com/apache/datafusion/issues/16306 for the DataFusion usecase)

Here is an example program.

```rust
use object_store::aws::AmazonS3Builder;
use object_store::{ObjectStore};
use object_store::path::Path;

#[tokio::main]
async fn main() {
// Goal is to read a parquet file from S3 without having to specify the region
// s3://clickhouse-public-datasets/hits_compatible/athena_partitioned/hits_1.parquet
let path = "hits_compatible/athena_partitioned/hits_1.parquet";

let store = AmazonS3Builder::new()
.with_bucket_name("clickhouse-public-datasets")
.with_skip_signature(true)
// NOTE the region is not specified. If it is set to `eu-central-1`
// this example works great, but if it is not set, it fails with:
// ```
// Error getting object from S3: Generic S3 error: Error performing in 120.898541ms -
// Received redirect without LOCATION, this normally indicates an incorrectly configured region
// ```
//.with_region("eu-central-1")
.build()
.expect("Failed to create S3 object store");

let path = Path::from(path);
let result = match store.
get(&path)
.await {
Ok(result) => result,
Err(e) => {
eprintln!("Error getting object from S3: {e}");
return;
}
};
println!("Successfully retrieved object from S3");
println!("Result meta: {:#?} ", result.meta);
}

```

**Describe the solution you'd like**
I want the request to succeed without having to explicitly specify the region.

**Describe alternatives you've considered**

Using `curl` you can see the response from AWS actually includes the correct region in the `x-amz-bucket-region` header

> < x-amz-bucket-region: eu-central-1

```shell
curl -v https://s3.us-east-1.amazonaws.com/clickhouse-public-datasets/hits_compatible/athena_partitioned/hits_1.parquet
...
* using HTTP/1.x
> GET /clickhouse-public-datasets/hits_compatible/athena_partitioned/hits_1.parquet HTTP/1.1
> Host: s3.us-east-1.amazonaws.com
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 301 Moved Permanently
< x-amz-bucket-region: eu-central-1
< x-amz-request-id: SG3WAD43HD3NADYW
< x-amz-id-2: qKq7zFq+trkB04Ie5CrBIZmYBMS5Lk4gVYH6Pq5NrO1rC+pmTPG8bxelOM3y3vRcweQiS1pBDwU=
< Content-Type: application/xml
< Transfer-Encoding: chunked
< Date: Fri, 06 Jun 2025 14:35:38 GMT
< Server: AmazonS3
<

* Connection #0 to host s3.us-east-1.amazonaws.com left intact
PermanentRedirectThe bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.clickhouse-public-datasets.s3.eu-central-1.amazonaws.comclickhouse-public-datasetsSG3WAD43HD3NADYWqKq7zFq+trkB04Ie5CrBIZmYBMS5Lk4gVYH6Pq5NrO1rC+pmTPG8bxelOM3y3vRcweQiS1pBDwU=
```

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start at the AmazonS3Builder configuration and the get call shown in the example, then trace how redirects and the x-amz-bucket-region response header are handled. Done means the example retrieves the S3 object without explicitly specifying a region and no longer fails on the redirect response.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, rust
Domain
backend, cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.