destination-s3-data-lake (Iceberg): Add explicit retry/backoff logic for AWS SDK operations
- Lenguaje dominante
- Python
- Estrellas
- 22.1k
- Forks
- 5.3k
- Métricas de merge de PR
- Métricas de PR pendientes
Descripción
## Summary
The Iceberg destination (destination-s3-data-lake) does not leverage AWS SDK's `BackoffStrategy` (e.g., `exponentialDelayWithoutJitter`) for retry/backoff logic. This could lead to transient failures not being handled gracefully, especially under load or during AWS service disruptions.
**Requested by:** @tgonzalezc5 (teo@airbyte.io)
## Investigation Findings
### 1. Explicit `.noRetry()` in PartitionedWriters
In `airbyte-cdk/bulk/toolkits/load-iceberg-parquet/src/main/kotlin/io/airbyte/cdk/load/toolkits/iceberg/parquet/io/PartitionedWriters.kt` (lines 67-70), the Iceberg `Tasks` utility is used with `.noRetry()` explicitly set:
```kotlin
Tasks.foreach(writers.values)
.throwFailureWhenFinished()
.noRetry()
.run(RowDataDeltaWriter::close, IOException::class.java)
```
### 2. No Retry Configuration on AWS SDK Clients
The S3 client in `airbyte-cdk/bulk/toolkits/load-s3/src/main/kotlin/io/airbyte/cdk/load/file/s3/S3Client.kt` (lines 224-252) is created without any explicit retry policy or backoff strategy:
```kotlin
val s3SdkClient =
aws.sdk.kotlin.services.s3.S3Client {
region = bucketConfig.s3BucketConfiguration.s3BucketRegion
credentialsProvider = credsProvider
endpointUrl = ...
httpClient(CrtHttpEngine)
forcePathStyle = true
}
```
### 3. No BackoffStrategy Imports
A search for `BackoffStrategy`, `exponentialDelay`, `RetryPolicy`, and `RetryStrategy` across the destination-s3-data-lake connector and load-iceberg-parquet toolkit returned no matches.
### 4. STS/Glue Clients Also Lack Retry Config
The `GlueCredentialsProvider` in `destination-s3-data-lake` creates STS clients without retry configuration.
## Recommendation
Consider adding explicit backoff strategies using AWS SDK's retry mechanisms, such as:
```kotlin
import software.amazon.awssdk.retries.api.BackoffStrategy
// Example: exponential backoff without jitter
val backoffStrategy = BackoffStrategy.exponentialDelayWithoutJitter(
Duration.ofMillis(100), // base delay
Duration.ofSeconds(20) // max delay
)
```
Reference: [AWS SDK BackoffStrategy](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/retries/api/BackoffStrategy.html#exponentialDelayWithoutJitter(java.time.Duration,java.time.Duration))
## Areas to Address
1. **S3 Client** (`load-s3` toolkit) - Add retry configuration to the Kotlin S3 client
2. **Glue/STS Clients** (`destination-s3-data-lake`) - Add retry configuration for credential providers
3. **Iceberg Tasks** (`load-iceberg-parquet`) - Evaluate whether `.noRetry()` is appropriate or if retry logic should be added
## Impact
Without proper retry/backoff logic, the destination may fail on transient errors that could otherwise be recovered from, leading to:
- Unnecessary sync failures
- Poor user experience during AWS service disruptions
- Potential data loss if partial writes aren't retried
---
*This issue was created based on an investigation by Devin.*
---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/10934
Guía de contribución
Evaluación
Este issue todavía no se ha evaluado.