airbytehq / airbytehq/airbyte

destination-s3-data-lake (Iceberg): Add explicit retry/backoff logic for AWS SDK operations

Abierto
#71,788 3 comentarios 0 reacciones 0 asignados Ver en GitHub
autoteam connectors/destination/s3-data-lake team/destinations
Lenguaje dominante
Python
Estrellas
22.1k
Forks
5.3k
Métricas de merge de PR
Métricas de PR pendientes

Descripción

## Summary

The Iceberg destination (destination-s3-data-lake) does not leverage AWS SDK's `BackoffStrategy` (e.g., `exponentialDelayWithoutJitter`) for retry/backoff logic. This could lead to transient failures not being handled gracefully, especially under load or during AWS service disruptions.

**Requested by:** @tgonzalezc5 (teo@airbyte.io)

## Investigation Findings

### 1. Explicit `.noRetry()` in PartitionedWriters

In `airbyte-cdk/bulk/toolkits/load-iceberg-parquet/src/main/kotlin/io/airbyte/cdk/load/toolkits/iceberg/parquet/io/PartitionedWriters.kt` (lines 67-70), the Iceberg `Tasks` utility is used with `.noRetry()` explicitly set:

```kotlin
Tasks.foreach(writers.values)
.throwFailureWhenFinished()
.noRetry()
.run(RowDataDeltaWriter::close, IOException::class.java)
```

### 2. No Retry Configuration on AWS SDK Clients

The S3 client in `airbyte-cdk/bulk/toolkits/load-s3/src/main/kotlin/io/airbyte/cdk/load/file/s3/S3Client.kt` (lines 224-252) is created without any explicit retry policy or backoff strategy:

```kotlin
val s3SdkClient =
aws.sdk.kotlin.services.s3.S3Client {
region = bucketConfig.s3BucketConfiguration.s3BucketRegion
credentialsProvider = credsProvider
endpointUrl = ...
httpClient(CrtHttpEngine)
forcePathStyle = true
}
```

### 3. No BackoffStrategy Imports

A search for `BackoffStrategy`, `exponentialDelay`, `RetryPolicy`, and `RetryStrategy` across the destination-s3-data-lake connector and load-iceberg-parquet toolkit returned no matches.

### 4. STS/Glue Clients Also Lack Retry Config

The `GlueCredentialsProvider` in `destination-s3-data-lake` creates STS clients without retry configuration.

## Recommendation

Consider adding explicit backoff strategies using AWS SDK's retry mechanisms, such as:

```kotlin
import software.amazon.awssdk.retries.api.BackoffStrategy

// Example: exponential backoff without jitter
val backoffStrategy = BackoffStrategy.exponentialDelayWithoutJitter(
Duration.ofMillis(100), // base delay
Duration.ofSeconds(20) // max delay
)
```

Reference: [AWS SDK BackoffStrategy](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/retries/api/BackoffStrategy.html#exponentialDelayWithoutJitter(java.time.Duration,java.time.Duration))

## Areas to Address

1. **S3 Client** (`load-s3` toolkit) - Add retry configuration to the Kotlin S3 client
2. **Glue/STS Clients** (`destination-s3-data-lake`) - Add retry configuration for credential providers
3. **Iceberg Tasks** (`load-iceberg-parquet`) - Evaluate whether `.noRetry()` is appropriate or if retry logic should be added

## Impact

Without proper retry/backoff logic, the destination may fail on transient errors that could otherwise be recovered from, leading to:
- Unnecessary sync failures
- Poor user experience during AWS service disruptions
- Potential data loss if partial writes aren't retried

---
*This issue was created based on an investigation by Devin.*

---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/10934

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.