airbytehq / airbytehq/airbyte

destination-s3-data-lake (Iceberg): Add explicit retry/backoff logic for AWS SDK operations

未關閉
#71,788 3 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
autoteam connectors/destination/s3-data-lake team/destinations
主要語言
Python
星號
22.1k
分支
5.4k
平均合併
5 小時
30 天內合併 PR
671

描述

## Summary

The Iceberg destination (destination-s3-data-lake) does not leverage AWS SDK's `BackoffStrategy` (e.g., `exponentialDelayWithoutJitter`) for retry/backoff logic. This could lead to transient failures not being handled gracefully, especially under load or during AWS service disruptions.

**Requested by:** @tgonzalezc5 (teo@airbyte.io)

## Investigation Findings

### 1. Explicit `.noRetry()` in PartitionedWriters

In `airbyte-cdk/bulk/toolkits/load-iceberg-parquet/src/main/kotlin/io/airbyte/cdk/load/toolkits/iceberg/parquet/io/PartitionedWriters.kt` (lines 67-70), the Iceberg `Tasks` utility is used with `.noRetry()` explicitly set:

```kotlin
Tasks.foreach(writers.values)
.throwFailureWhenFinished()
.noRetry()
.run(RowDataDeltaWriter::close, IOException::class.java)
```

### 2. No Retry Configuration on AWS SDK Clients

The S3 client in `airbyte-cdk/bulk/toolkits/load-s3/src/main/kotlin/io/airbyte/cdk/load/file/s3/S3Client.kt` (lines 224-252) is created without any explicit retry policy or backoff strategy:

```kotlin
val s3SdkClient =
aws.sdk.kotlin.services.s3.S3Client {
region = bucketConfig.s3BucketConfiguration.s3BucketRegion
credentialsProvider = credsProvider
endpointUrl = ...
httpClient(CrtHttpEngine)
forcePathStyle = true
}
```

### 3. No BackoffStrategy Imports

A search for `BackoffStrategy`, `exponentialDelay`, `RetryPolicy`, and `RetryStrategy` across the destination-s3-data-lake connector and load-iceberg-parquet toolkit returned no matches.

### 4. STS/Glue Clients Also Lack Retry Config

The `GlueCredentialsProvider` in `destination-s3-data-lake` creates STS clients without retry configuration.

## Recommendation

Consider adding explicit backoff strategies using AWS SDK's retry mechanisms, such as:

```kotlin
import software.amazon.awssdk.retries.api.BackoffStrategy

// Example: exponential backoff without jitter
val backoffStrategy = BackoffStrategy.exponentialDelayWithoutJitter(
Duration.ofMillis(100), // base delay
Duration.ofSeconds(20) // max delay
)
```

Reference: [AWS SDK BackoffStrategy](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/retries/api/BackoffStrategy.html#exponentialDelayWithoutJitter(java.time.Duration,java.time.Duration))

## Areas to Address

1. **S3 Client** (`load-s3` toolkit) - Add retry configuration to the Kotlin S3 client
2. **Glue/STS Clients** (`destination-s3-data-lake`) - Add retry configuration for credential providers
3. **Iceberg Tasks** (`load-iceberg-parquet`) - Evaluate whether `.noRetry()` is appropriate or if retry logic should be added

## Impact

Without proper retry/backoff logic, the destination may fail on transient errors that could otherwise be recovered from, leading to:
- Unnecessary sync failures
- Poor user experience during AWS service disruptions
- Potential data loss if partial writes aren't retried

---
*This issue was created based on an investigation by Devin.*

---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/10934

貢獻指南

開啟貢獻指南

研究方向

Start with S3Client.kt in the load-s3 toolkit and inspect how the Kotlin S3 client is configured. Then review GlueCredentialsProvider and PartitionedWriters.kt, focusing on the missing retry configuration and the explicit noRetry() call. Done means the relevant AWS operations and Iceberg writer behavior have an agreed, tested retry/backoff approach.

由索引模型根據 Issue 內容生成。

評估

技術堆疊
aws, kotlin
領域
cloud, data-engineering
Issue 類型
功能
難度
4/5
預估耗時
3-5 天
活躍度
停滯
描述清晰度
基本清楚
新手友好度
35/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。