airbytehq / airbytehq/airbyte

destination-s3-data-lake: 403 s3:ListBucket with STS-scoped vended credentials (missing trailing slash in IcebergTableCleaner)

Offen
#81,442 2 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
autoteam community connectors/destination/s3-data-lake team/use
Vorherrschende Sprache
Python
Sterne
22.1k
Forks
5.3k
PR-Merge-Kennzahlen
PR-Kennzahlen ausstehend

Beschreibung

## Connector
`destination-s3-data-lake` (Load CDK `load-iceberg-parquet` toolkit), REST/POLARIS catalog with **STS credential vending** (e.g. Lakekeeper, or any Iceberg REST catalog that returns scoped temporary AWS credentials in `loadTable`).

## What happened
A sync (or connection CHECK that triggers table cleanup) fails with:

```
software.amazon.awssdk.services.s3.model.S3Exception (AccessDenied):
User: arn:aws:sts:::assumed-role/ is not authorized to perform:
s3:ListBucket on resource: "arn:aws:s3:::" ...
at org.apache.iceberg.aws.s3.S3FileIO.deletePrefix(S3FileIO.java:365)
at io.airbyte.cdk.load.toolkits.iceberg.parquet.io.IcebergTableCleaner.clearTable(IcebergTableCleaner.kt:41)
```

## Root cause
`IcebergTableCleaner.clearTable()` passes the table location **without a trailing slash** to `SupportsPrefixOperations.deletePrefix()`, which forwards it into `S3FileIO.listPrefix()` → `ListObjectsV2(prefix=)`.

Iceberg REST catalogs commonly vend STS credentials scoped with a session policy like:

```json
{ "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::",
"Condition": { "StringLike": { "s3:prefix": "///*" } } }
```

`StringLike` with `.../*` requires the request prefix to include the `/` before the wildcard boundary. A prefix of `<...>/` (no trailing slash) does not satisfy the condition, so `ListObjectsV2` is denied with 403.

**Verified manually:**
- `ListObjectsV2(prefix="...//")` → 200
- `ListObjectsV2(prefix=".../")` → 403 AccessDenied

Secondary correctness issue: without the trailing slash, `deletePrefix` can also match sibling tables that share a name prefix (`orders` matching `orders_archive/...`).

## Expected
Table cleanup should scope the prefix to the table directory (trailing slash), matching what Iceberg's own `FileSystemWalker` already does before calling `listPrefix`, and what Trino's `S3FileIO` usage does via `toDirectoryPath()`.

## Proposed fix
One line in `IcebergTableCleaner.clearTable`:
```kotlin
val prefix = if (tableLocation.endsWith("/")) tableLocation else "$tableLocation/"
io.deletePrefix(prefix)
```
Idempotent (handles already-slashed input) and prevents sibling-prefix bleed. PR: #78624.

## Validation
Built a custom connector image with this patch and confirmed end-to-end against Lakekeeper with STS credential vending: the 403 disappears and a full sync writes to the Iceberg table successfully. The stock connector (unpatched) fails the CHECK with the 403 above on the same setup.

---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/13058

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.