[STORAGE-S3] Add Support for Custom S3 Endpoint
- Lingua principale
- Python
- Stelle
- 22.1k
- Fork
- 5.3k
- Metriche di merge delle PR
- Metriche PR in attesa
Descrizione
### Helm Chart Version
0.429.0
### What step the error happened?
Upgrading the Platform or Helm Chart
### Relevant information
## Issue: Add Support for Custom S3 Endpoint in `log4j2-s3.xml`
### Description
Currently, the log4j2-s3.xml configuration file in the Airbyte platform is hardcoded to work with Amazon S3 as the default storage option. However, when deploying Airbyte via the Helm chart, it's not possible to override the default AWS endpoint (amazonaws.com) to use S3-compatible storage services from other providers (e.g., Scaleway).
To illustrate, it's not possible to set an endpoint like https://s3.fr-par.scw.cloud for Scaleway’s S3-compatible storage service.
For instance, it is not currently possible to use an endpoint like `https://s3.fr-par.scw.cloud`.
### Proposal
To enhance compatibility with other S3-compatible storage services, I propose adding support for a custom S3 endpoint in the log4j2-s3.xml file. Below is a suggested change:
```xml
${sys:ciMode:-false}
%d{yyyy-MM-dd HH:mm:ss,SSS}{GMT+0} [dd.trace_id=%X{dd.trace_id} dd.span_id=%X{dd.span_id}] %p %C{1.}(%M):%L %replace{%m}{apikey=[\w\-]*}{apikey=*****}%n
%d{yyyy-MM-dd HH:mm:ss}{GMT+0} %highlight{%p} %C{1.}(%M):%L - %replace{%m}{apikey=[\w\-]*}{apikey=*****}%n
%d{yyyy-MM-dd HH:mm:ss}{GMT+0}%replace{ %X{log_source}}{^ -}{} > %replace{%m}{apikey=[\w\-]*}{apikey=*****}%n
${sys:LOG_LEVEL:-${env:LOG_LEVEL:-INFO}}
${env:LOG_IDLE_ROUTE_TTL:-15}
${sys:STORAGE_BUCKET_LOG:-${env:STORAGE_BUCKET_LOG:-}}
${sys:AWS_DEFAULT_REGION:-${env:AWS_DEFAULT_REGION:-}}
${sys:AWS_ENDPOINT_URL:-${env:AWS_ENDPOINT_URL:-}}
```
In addition to modifying the log4j2-s3.xml file, a corresponding change would be necessary in the `values.yaml` file within the `Helm chart` to support custom S3 endpoints. The change should be made in the global.storage section, as shown below:
```yaml
storage:
# -- The storage backend type. Supports s3, gcs, minio (default)
type: s3 # change to your preferred storage type
# -- Secret name where storage provider credentials are stored
#storageSecretName: "airbyte-config-secrets"
# S3
bucket: ## S3 bucket names that you've created. We recommend storing the following all in one bucket.
log: airbyte-bucket
state: airbyte-bucket
workloadOutput: airbyte-bucket
s3:
region: "" ## e.g. us-east-1
authenticationType: credentials ## Use "credentials" or "instanceProfile"
#############################################
# Add the following to support custom S3 endpoint:
endpoint:
#############################################
# GCS
#bucket: ## GCS bucket names that you've created. We recommend storing the following all in one bucket.
# log: airbyte-bucket
# state: airbyte-bucket
# workloadOutput: airbyte-bucket
#gcs:
# projectId:
# credentialsJson: /secrets/gcs-log-creds/gcp.json
```
#### Expected Outcome
These changes will allow users to specify an alternative S3-compatible endpoint when configuring Airbyte with non-AWS S3 storage providers. This enhancement will improve the flexibility and compatibility of Airbyte, enabling its deployment in a broader range of cloud storage environments.
### Relevant log output
_No response_
Guida per i contributori
Apri la guida per i contributori
Valutazione
Questa issue non è ancora stata valutata.