[destination-s3-data-lake] main_branch_name is ignored for Nessie; all commits go to main
- Langage dominant
- Python
- Étoiles
- 22.1k
- Forks
- 5.3k
- Métriques de merge des PR
- Métriques de PR en attente
Description
### Connector Name
destination-s3-data-lake
### Connector Version
0.3.52
### What step the error happened?
During the sync
### Relevant information
We are running Airbyte 2.0.1 with the `destination-s3-data-lake` connector against a Nessie catalog. We configured `main_branch_name: staging` and confirmed the config is passed through to the connector pod. The `staging` branch exists in Nessie. Still, every sync and destination test commits to `main`.
We traced this to `S3DataLakeUtil.kt`. The connector currently sets:
```
NessieConfigConstants.CONF_NESSIE_REF to config.icebergCatalogConfiguration.mainBranchName
```
This produces the property key `"nessie.ref"`. But Iceberg's NessieCatalog.initialize() strips the `"nessie."` prefix and reads `"ref"`:
```kt
String branch = properties.get(CatalogProperties.REF);
```
So the configured branch is never picked up and the catalog defaults to `main`.
### Steps to reproduce:
1. Set up a Nessie destination with `main_branch_name = "staging"`.
2. Make sure `staging` exists in Nessie.
3. Run a sync.
4. Check Nessie and sync logs — all commits use `ref=main`.
### Expected behavior:
Commits should go to the configured `main_branch_name`.
### What worked for us:
We patched the connector locally by changing the property key to the literal string `"ref"`:
```
"ref" to config.icebergCatalogConfiguration.mainBranchName
```
We also updated `S3DataLakeUtilTest.kt` to assert `catalogProperties["ref"]` instead of `catalogProperties["nessie.ref"]`. After deploying the patched image, our syncs started committing to `staging` as expected.
We are not sure if this is the intended fix upstream, but it solved the issue in our environment.
### Relevant log output
```shell
Observed in Airbyte sync logs — every commit goes to Branch{name=main} regardless of the configured main_branch_name:
INFO NessieIcebergClient(commitContent):698 Committed '.' against 'Branch{name=main, metadata=null, hash=}', expected commit-id was ''
This pattern repeats for every table in the sync. The configured branch name is never applied.
```
### Contribute
- [x] Yes, I want to contribute
---
**Internal Tracking:** airbytehq/oncall#13151
Guide de contribution
Ouvrir le guide de contribution
Évaluation
Cette issue n'a pas encore été évaluée.