apache / apache/arrow

[R] Cannot override environmental variables setting AWS S3 region

Open
#32,493 0 comments 0 reactions 0 assignees View on GitHub
Component: R Type: task
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

If a user has set the AWS_DEFAULT_REGION, there is no way we can override that setting (especially to a null value) in a one-off call.  Consider the following example.  This fails:

```java

library(arrow)
Sys.setenv(AWS_DEFAULT_REGION="data")

noaa <- s3_bucket("neon4cast-drivers/",
                  endpoint_override = "data.ecoforecast.org",
                  anonymous = TRUE)
```
If the env var is not set or unset, this succeeds.  However, attempting to override the region does not help:

```java

noaa <- s3_bucket("neon4cast-drivers/",
                  endpoint_override = "data.ecoforecast.org",
region = "us-east-1",
                  anonymous = TRUE)
```
(nor does it help to set the region to "", or NULL. – note this is a MINIO host and so region is not needed anyway). 

Relatedly, one might expect that AWS_S3_ENDPOINT could be used instead of setting `endpoint_override`, but this does not seem to work either (lemme know if you want that in a separate issue thread).  At least on the plus side, the above code does not fail if AWS_S3_ENDPOINT is set to some other value than given in endpoint_override, so that is nice). 

**Reporter**: [Carl Boettiger](https://issues.apache.org/jira/browse/ARROW-17201) / @cboettig

**Note**: *This issue was originally created as [ARROW-17201](https://issues.apache.org/jira/browse/ARROW-17201). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the R examples with AWS_DEFAULT_REGION set, then trace how endpoint_override, region, and environment variables are handled for S3 connections. Done means a one-off call can override or clear the environment-provided region when using the shown MINIO endpoint, with the reported examples succeeding.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, r
Domain
cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.