airbytehq / airbytehq/airbyte

[STORAGE-S3] Add Support for Custom S3 Endpoint

Aperta
#44,174 21 commenti 14 reazioni 0 assegnatari Vedi su GitHub
area/platform autoteam community logs team/deployments type/bug
Lingua principale
Python
Stelle
22.1k
Fork
5.3k
Metriche di merge delle PR
Metriche PR in attesa

Descrizione

### Helm Chart Version

0.429.0

### What step the error happened?

Upgrading the Platform or Helm Chart

### Relevant information

## Issue: Add Support for Custom S3 Endpoint in `log4j2-s3.xml`

### Description

Currently, the log4j2-s3.xml configuration file in the Airbyte platform is hardcoded to work with Amazon S3 as the default storage option. However, when deploying Airbyte via the Helm chart, it's not possible to override the default AWS endpoint (amazonaws.com) to use S3-compatible storage services from other providers (e.g., Scaleway).

To illustrate, it's not possible to set an endpoint like https://s3.fr-par.scw.cloud for Scaleway’s S3-compatible storage service.

For instance, it is not currently possible to use an endpoint like `https://s3.fr-par.scw.cloud`.

### Proposal

To enhance compatibility with other S3-compatible storage services, I propose adding support for a custom S3 endpoint in the log4j2-s3.xml file. Below is a suggested change:

```xml

${sys:ciMode:-false}

%d{yyyy-MM-dd HH:mm:ss,SSS}{GMT+0} [dd.trace_id=%X{dd.trace_id} dd.span_id=%X{dd.span_id}] %p %C{1.}(%M):%L %replace{%m}{apikey=[\w\-]*}{apikey=*****}%n

%d{yyyy-MM-dd HH:mm:ss}{GMT+0} %highlight{%p} %C{1.}(%M):%L - %replace{%m}{apikey=[\w\-]*}{apikey=*****}%n

%d{yyyy-MM-dd HH:mm:ss}{GMT+0}%replace{ %X{log_source}}{^ -}{} > %replace{%m}{apikey=[\w\-]*}{apikey=*****}%n


${sys:LOG_LEVEL:-${env:LOG_LEVEL:-INFO}}

${env:LOG_IDLE_ROUTE_TTL:-15}


${sys:STORAGE_BUCKET_LOG:-${env:STORAGE_BUCKET_LOG:-}}
${sys:AWS_DEFAULT_REGION:-${env:AWS_DEFAULT_REGION:-}}

${sys:AWS_ENDPOINT_URL:-${env:AWS_ENDPOINT_URL:-}}

```
In addition to modifying the log4j2-s3.xml file, a corresponding change would be necessary in the `values.yaml` file within the `Helm chart` to support custom S3 endpoints. The change should be made in the global.storage section, as shown below:

```yaml
storage:
# -- The storage backend type. Supports s3, gcs, minio (default)
type: s3 # change to your preferred storage type
# -- Secret name where storage provider credentials are stored
#storageSecretName: "airbyte-config-secrets"

# S3
bucket: ## S3 bucket names that you've created. We recommend storing the following all in one bucket.
log: airbyte-bucket
state: airbyte-bucket
workloadOutput: airbyte-bucket
s3:
region: "" ## e.g. us-east-1
authenticationType: credentials ## Use "credentials" or "instanceProfile"

#############################################
# Add the following to support custom S3 endpoint:
endpoint:
#############################################

# GCS
#bucket: ## GCS bucket names that you've created. We recommend storing the following all in one bucket.
# log: airbyte-bucket
# state: airbyte-bucket
# workloadOutput: airbyte-bucket
#gcs:
# projectId:
# credentialsJson: /secrets/gcs-log-creds/gcp.json

```

#### Expected Outcome

These changes will allow users to specify an alternative S3-compatible endpoint when configuring Airbyte with non-AWS S3 storage providers. This enhancement will improve the flexibility and compatibility of Airbyte, enabling its deployment in a broader range of cloud storage environments.

### Relevant log output

_No response_

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.