apache / apache/airflow

Support for vhost-style URL for S3

Open
#54,918 2 comments 1 reaction 0 assignees View on GitHub
area:providers kind:feature needs-triage provider:amazon
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Description

Hi! I use the Airflow feature to store logs on S3.

I am currently using the `path-style` by setting the `endpoint_url` separately in the connection parameters and the bucket name in the variable `AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER`.

I also have to set `AWS_S3_PATH_STYLE_ACCESS="true"` for the boto3 library.

So my settings look like this

````
export AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER="s3://my-bucket"

airflow connections add my-conn --conn-type aws --conn-extra "{\
...
\"endpoint_url\": \"https://s3.example.com\",
}"
````

I would prefer to use a `vhost-style` URL, for example "https://my-bucket.s3.example.com" but I can't find a way to configure Airflow to use S3 in this mode.

### Use case/motivation

_No response_

### Related issues

_No response_

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start by tracing how AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER and the AWS connection's endpoint_url are passed to the S3 logging integration, including the existing AWS_S3_PATH_STYLE_ACCESS setting used by boto3. Check whether the current configuration and documentation expose a way to select virtual-hosted-style URLs; done means a supported configuration for the requested https://my-bucket.s3.example.com form with coverage for the existing path-style behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.