destination-s3-data-lake: 403 s3:ListBucket with STS-scoped vended credentials (missing trailing slash in IcebergTableCleaner)
- Vorherrschende Sprache
- Python
- Sterne
- 22.1k
- Forks
- 5.3k
- PR-Merge-Kennzahlen
- PR-Kennzahlen ausstehend
Beschreibung
## Connector
`destination-s3-data-lake` (Load CDK `load-iceberg-parquet` toolkit), REST/POLARIS catalog with **STS credential vending** (e.g. Lakekeeper, or any Iceberg REST catalog that returns scoped temporary AWS credentials in `loadTable`).
## What happened
A sync (or connection CHECK that triggers table cleanup) fails with:
```
software.amazon.awssdk.services.s3.model.S3Exception (AccessDenied):
User: arn:aws:sts:::assumed-role/ is not authorized to perform:
s3:ListBucket on resource: "arn:aws:s3:::" ...
at org.apache.iceberg.aws.s3.S3FileIO.deletePrefix(S3FileIO.java:365)
at io.airbyte.cdk.load.toolkits.iceberg.parquet.io.IcebergTableCleaner.clearTable(IcebergTableCleaner.kt:41)
```
## Root cause
`IcebergTableCleaner.clearTable()` passes the table location **without a trailing slash** to `SupportsPrefixOperations.deletePrefix()`, which forwards it into `S3FileIO.listPrefix()` → `ListObjectsV2(prefix=)`.
Iceberg REST catalogs commonly vend STS credentials scoped with a session policy like:
```json
{ "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::",
"Condition": { "StringLike": { "s3:prefix": "///*" } } }
```
`StringLike` with `.../*` requires the request prefix to include the `/` before the wildcard boundary. A prefix of `<...>/` (no trailing slash) does not satisfy the condition, so `ListObjectsV2` is denied with 403.
**Verified manually:**
- `ListObjectsV2(prefix="...//")` → 200
- `ListObjectsV2(prefix=".../")` → 403 AccessDenied
Secondary correctness issue: without the trailing slash, `deletePrefix` can also match sibling tables that share a name prefix (`orders` matching `orders_archive/...`).
## Expected
Table cleanup should scope the prefix to the table directory (trailing slash), matching what Iceberg's own `FileSystemWalker` already does before calling `listPrefix`, and what Trino's `S3FileIO` usage does via `toDirectoryPath()`.
## Proposed fix
One line in `IcebergTableCleaner.clearTable`:
```kotlin
val prefix = if (tableLocation.endsWith("/")) tableLocation else "$tableLocation/"
io.deletePrefix(prefix)
```
Idempotent (handles already-slashed input) and prevents sibling-prefix bleed. PR: #78624.
## Validation
Built a custom connector image with this patch and confirmed end-to-end against Lakekeeper with STS credential vending: the 403 disappears and a full sync writes to the Iceberg table successfully. The stock connector (unpatched) fails the CHECK with the 403 above on the same setup.
---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/13058
Beitragsleitfaden
Bewertung
Dieses Issue wurde noch nicht bewertet.