destination-s3-data-lake (Iceberg): Add explicit retry/backoff logic for AWS SDK operations
- 主要語言
- Python
- 星號
- 22.1k
- 分支
- 5.4k
- 平均合併
- 5 小時
- 30 天內合併 PR
- 671
描述
## Summary
The Iceberg destination (destination-s3-data-lake) does not leverage AWS SDK's `BackoffStrategy` (e.g., `exponentialDelayWithoutJitter`) for retry/backoff logic. This could lead to transient failures not being handled gracefully, especially under load or during AWS service disruptions.
**Requested by:** @tgonzalezc5 (teo@airbyte.io)
## Investigation Findings
### 1. Explicit `.noRetry()` in PartitionedWriters
In `airbyte-cdk/bulk/toolkits/load-iceberg-parquet/src/main/kotlin/io/airbyte/cdk/load/toolkits/iceberg/parquet/io/PartitionedWriters.kt` (lines 67-70), the Iceberg `Tasks` utility is used with `.noRetry()` explicitly set:
```kotlin
Tasks.foreach(writers.values)
.throwFailureWhenFinished()
.noRetry()
.run(RowDataDeltaWriter::close, IOException::class.java)
```
### 2. No Retry Configuration on AWS SDK Clients
The S3 client in `airbyte-cdk/bulk/toolkits/load-s3/src/main/kotlin/io/airbyte/cdk/load/file/s3/S3Client.kt` (lines 224-252) is created without any explicit retry policy or backoff strategy:
```kotlin
val s3SdkClient =
aws.sdk.kotlin.services.s3.S3Client {
region = bucketConfig.s3BucketConfiguration.s3BucketRegion
credentialsProvider = credsProvider
endpointUrl = ...
httpClient(CrtHttpEngine)
forcePathStyle = true
}
```
### 3. No BackoffStrategy Imports
A search for `BackoffStrategy`, `exponentialDelay`, `RetryPolicy`, and `RetryStrategy` across the destination-s3-data-lake connector and load-iceberg-parquet toolkit returned no matches.
### 4. STS/Glue Clients Also Lack Retry Config
The `GlueCredentialsProvider` in `destination-s3-data-lake` creates STS clients without retry configuration.
## Recommendation
Consider adding explicit backoff strategies using AWS SDK's retry mechanisms, such as:
```kotlin
import software.amazon.awssdk.retries.api.BackoffStrategy
// Example: exponential backoff without jitter
val backoffStrategy = BackoffStrategy.exponentialDelayWithoutJitter(
Duration.ofMillis(100), // base delay
Duration.ofSeconds(20) // max delay
)
```
Reference: [AWS SDK BackoffStrategy](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/retries/api/BackoffStrategy.html#exponentialDelayWithoutJitter(java.time.Duration,java.time.Duration))
## Areas to Address
1. **S3 Client** (`load-s3` toolkit) - Add retry configuration to the Kotlin S3 client
2. **Glue/STS Clients** (`destination-s3-data-lake`) - Add retry configuration for credential providers
3. **Iceberg Tasks** (`load-iceberg-parquet`) - Evaluate whether `.noRetry()` is appropriate or if retry logic should be added
## Impact
Without proper retry/backoff logic, the destination may fail on transient errors that could otherwise be recovered from, leading to:
- Unnecessary sync failures
- Poor user experience during AWS service disruptions
- Potential data loss if partial writes aren't retried
---
*This issue was created based on an investigation by Devin.*
---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/10934
貢獻指南
研究方向
Start with S3Client.kt in the load-s3 toolkit and inspect how the Kotlin S3 client is configured. Then review GlueCredentialsProvider and PartitionedWriters.kt, focusing on the missing retry configuration and the explicit noRetry() call. Done means the relevant AWS operations and Iceberg writer behavior have an agreed, tested retry/backoff approach.
由索引模型根據 Issue 內容生成。
評估
- 技術堆疊
- aws, kotlin
- 領域
- cloud, data-engineering
- Issue 類型
- 功能
- 難度
- 4/5
- 預估耗時
- 3-5 天
- 活躍度
- 停滯
- 描述清晰度
- 基本清楚
- 新手友好度
- 35/100