destination-s3-data-lake: cannot write to nested (multi-level) Iceberg namespaces — dotted namespace treated as one level
- Vorherrschende Sprache
- Python
- Sterne
- 22.1k
- Forks
- 5.4k
- Ø Merge
- 5 Std.
- Gemergte PRs (30 T.)
- 671
Beschreibung
## Connector
`destination-s3-data-lake` (Load CDK `load-iceberg-parquet` toolkit), any Iceberg REST catalog that enforces multi-level namespaces (e.g. Lakekeeper, Polaris).
## What happened
Configuring a destination namespace with dots — e.g. `operational.inventory.bronze` (a multi-level Iceberg namespace) — fails at CHECK/sync:
```
org.apache.iceberg.exceptions.BadRequestException: Malformed request: Namespace parts cannot contain '.'
at ... (during table create/cleanup)
```
## Root cause
`TableIdGenerator.tableIdOf` builds the identifier from the raw namespace string as a **single** namespace level:
```kotlin
// airbyte-cdk/bulk/toolkits/load-iceberg-parquet/.../TableIdGenerator.kt
fun tableIdOf(namespace: String, name: String): TableIdentifier =
TableIdentifier.of(Namespace.of(namespace), name)
```
`Namespace.of("operational.inventory.bronze")` yields a namespace with **one** level whose text contains dots. Iceberg's spec treats `.` as the level separator, and REST catalogs reject a single level containing `.`. So any nested namespace target is unusable — the connector can only write to a single flat namespace.
## Expected
A dotted namespace should map to a multi-level Iceberg namespace (`operational` › `lims` › `inventory` › `bronze`), consistent with how Trino, PyIceberg, and the Iceberg REST spec interpret dotted names.
## Proposed fix
Split on `.` into levels:
```kotlin
fun tableIdOf(namespace: String, name: String): TableIdentifier =
TableIdentifier.of(Namespace.of(*namespace.split(".").toTypedArray()), name)
```
Single-level names are unaffected (`split(".")` on a dotless string returns a one-element array).
## Validation
Built a custom connector image with this fix and confirmed against Lakekeeper: CHECK and full syncs now create tables under the correct multi-level namespace (`operational.lims..bronze`), where the stock connector fails with the BadRequestException above. Replicated ~457k rows across 5 nested namespaces this way.
## Note
Related but independent from #81442 / PR #78624 (trailing-slash normalization in `IcebergTableCleaner`). Both are needed to use this connector with Lakekeeper-style REST catalogs.
---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/13062
Beitragsleitfaden
Rechercherichtung
Read airbyte-cdk/bulk/toolkits/load-iceberg-parquet/.../TableIdGenerator.kt and start at TableIdGenerator.tableIdOf. Run CHECK and a full sync against a REST catalog with a dotted namespace; done means tables are created under separate namespace levels without the malformed-request error.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Bewertung
- Tech-Stack
- aws, kotlin
- Bereich
- data-engineering
- Issue-Typ
- Bug
- Schwierigkeit
- 2/5
- Geschätzter Aufwand
- 1-3 Stunden
- Aktivitätsstatus
- Ruhig
- Klarheit
- Klar beschrieben
- Anfängerfreundlichkeit
- 72/100