airbytehq / airbytehq/airbyte

destination-s3-data-lake: cannot write to nested (multi-level) Iceberg namespaces — dotted namespace treated as one level

Offen Anfängerfreundlich
#81,450 2 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
autoteam community connectors/destination/s3-data-lake team/use
Vorherrschende Sprache
Python
Sterne
22.1k
Forks
5.4k
Ø Merge
5 Std.
Gemergte PRs (30 T.)
671

Beschreibung

## Connector
`destination-s3-data-lake` (Load CDK `load-iceberg-parquet` toolkit), any Iceberg REST catalog that enforces multi-level namespaces (e.g. Lakekeeper, Polaris).

## What happened
Configuring a destination namespace with dots — e.g. `operational.inventory.bronze` (a multi-level Iceberg namespace) — fails at CHECK/sync:

```
org.apache.iceberg.exceptions.BadRequestException: Malformed request: Namespace parts cannot contain '.'
at ... (during table create/cleanup)
```

## Root cause
`TableIdGenerator.tableIdOf` builds the identifier from the raw namespace string as a **single** namespace level:

```kotlin
// airbyte-cdk/bulk/toolkits/load-iceberg-parquet/.../TableIdGenerator.kt
fun tableIdOf(namespace: String, name: String): TableIdentifier =
TableIdentifier.of(Namespace.of(namespace), name)
```

`Namespace.of("operational.inventory.bronze")` yields a namespace with **one** level whose text contains dots. Iceberg's spec treats `.` as the level separator, and REST catalogs reject a single level containing `.`. So any nested namespace target is unusable — the connector can only write to a single flat namespace.

## Expected
A dotted namespace should map to a multi-level Iceberg namespace (`operational` › `lims` › `inventory` › `bronze`), consistent with how Trino, PyIceberg, and the Iceberg REST spec interpret dotted names.

## Proposed fix
Split on `.` into levels:

```kotlin
fun tableIdOf(namespace: String, name: String): TableIdentifier =
TableIdentifier.of(Namespace.of(*namespace.split(".").toTypedArray()), name)
```

Single-level names are unaffected (`split(".")` on a dotless string returns a one-element array).

## Validation
Built a custom connector image with this fix and confirmed against Lakekeeper: CHECK and full syncs now create tables under the correct multi-level namespace (`operational.lims..bronze`), where the stock connector fails with the BadRequestException above. Replicated ~457k rows across 5 nested namespaces this way.

## Note
Related but independent from #81442 / PR #78624 (trailing-slash normalization in `IcebergTableCleaner`). Both are needed to use this connector with Lakekeeper-style REST catalogs.

---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/13062

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Read airbyte-cdk/bulk/toolkits/load-iceberg-parquet/.../TableIdGenerator.kt and start at TableIdGenerator.tableIdOf. Run CHECK and a full sync against a REST catalog with a dotted namespace; done means tables are created under separate namespace levels without the malformed-request error.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
aws, kotlin
Bereich
data-engineering
Issue-Typ
Bug
Schwierigkeit
2/5
Geschätzter Aufwand
1-3 Stunden
Aktivitätsstatus
Ruhig
Klarheit
Klar beschrieben
Anfängerfreundlichkeit
72/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.