airbytehq / airbytehq/airbyte

[STORAGE-S3] Add Support for Custom S3 Endpoint

未關閉
#44,174 21 則留言 14 個 reaction 已指派 0 人 在 GitHub 檢視
area/platform autoteam community logs team/deployments type/bug
主要語言
Python
星號
22.1k
分支
5.4k
平均合併
5 小時
30 天內合併 PR
671

描述

### Helm Chart Version

0.429.0

### What step the error happened?

Upgrading the Platform or Helm Chart

### Relevant information

## Issue: Add Support for Custom S3 Endpoint in `log4j2-s3.xml`

### Description

Currently, the log4j2-s3.xml configuration file in the Airbyte platform is hardcoded to work with Amazon S3 as the default storage option. However, when deploying Airbyte via the Helm chart, it's not possible to override the default AWS endpoint (amazonaws.com) to use S3-compatible storage services from other providers (e.g., Scaleway).

To illustrate, it's not possible to set an endpoint like https://s3.fr-par.scw.cloud for Scaleway’s S3-compatible storage service.

For instance, it is not currently possible to use an endpoint like `https://s3.fr-par.scw.cloud`.

### Proposal

To enhance compatibility with other S3-compatible storage services, I propose adding support for a custom S3 endpoint in the log4j2-s3.xml file. Below is a suggested change:

```xml

${sys:ciMode:-false}

%d{yyyy-MM-dd HH:mm:ss,SSS}{GMT+0} [dd.trace_id=%X{dd.trace_id} dd.span_id=%X{dd.span_id}] %p %C{1.}(%M):%L %replace{%m}{apikey=[\w\-]*}{apikey=*****}%n

%d{yyyy-MM-dd HH:mm:ss}{GMT+0} %highlight{%p} %C{1.}(%M):%L - %replace{%m}{apikey=[\w\-]*}{apikey=*****}%n

%d{yyyy-MM-dd HH:mm:ss}{GMT+0}%replace{ %X{log_source}}{^ -}{} > %replace{%m}{apikey=[\w\-]*}{apikey=*****}%n


${sys:LOG_LEVEL:-${env:LOG_LEVEL:-INFO}}

${env:LOG_IDLE_ROUTE_TTL:-15}


${sys:STORAGE_BUCKET_LOG:-${env:STORAGE_BUCKET_LOG:-}}
${sys:AWS_DEFAULT_REGION:-${env:AWS_DEFAULT_REGION:-}}

${sys:AWS_ENDPOINT_URL:-${env:AWS_ENDPOINT_URL:-}}

```
In addition to modifying the log4j2-s3.xml file, a corresponding change would be necessary in the `values.yaml` file within the `Helm chart` to support custom S3 endpoints. The change should be made in the global.storage section, as shown below:

```yaml
storage:
# -- The storage backend type. Supports s3, gcs, minio (default)
type: s3 # change to your preferred storage type
# -- Secret name where storage provider credentials are stored
#storageSecretName: "airbyte-config-secrets"

# S3
bucket: ## S3 bucket names that you've created. We recommend storing the following all in one bucket.
log: airbyte-bucket
state: airbyte-bucket
workloadOutput: airbyte-bucket
s3:
region: "" ## e.g. us-east-1
authenticationType: credentials ## Use "credentials" or "instanceProfile"

#############################################
# Add the following to support custom S3 endpoint:
endpoint:
#############################################

# GCS
#bucket: ## GCS bucket names that you've created. We recommend storing the following all in one bucket.
# log: airbyte-bucket
# state: airbyte-bucket
# workloadOutput: airbyte-bucket
#gcs:
# projectId:
# credentialsJson: /secrets/gcs-log-creds/gcp.json

```

#### Expected Outcome

These changes will allow users to specify an alternative S3-compatible endpoint when configuring Airbyte with non-AWS S3 storage providers. This enhancement will improve the flexibility and compatibility of Airbyte, enabling its deployment in a broader range of cloud storage environments.

### Relevant log output

_No response_

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。