airbytehq / airbytehq/airbyte

[destination-s3-data-lake] main_branch_name is ignored for Nessie; all commits go to main

Ouverte
#82,732 2 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
area/connectors autoteam community connectors/destination/s3-data-lake needs-triage team/use type/bug
Langage dominant
Python
Étoiles
22.1k
Forks
5.3k
Métriques de merge des PR
Métriques de PR en attente

Description

### Connector Name

destination-s3-data-lake

### Connector Version

0.3.52

### What step the error happened?

During the sync

### Relevant information

We are running Airbyte 2.0.1 with the `destination-s3-data-lake` connector against a Nessie catalog. We configured `main_branch_name: staging` and confirmed the config is passed through to the connector pod. The `staging` branch exists in Nessie. Still, every sync and destination test commits to `main`.

We traced this to `S3DataLakeUtil.kt`. The connector currently sets:
```
NessieConfigConstants.CONF_NESSIE_REF to config.icebergCatalogConfiguration.mainBranchName
```

This produces the property key `"nessie.ref"`. But Iceberg's NessieCatalog.initialize() strips the `"nessie."` prefix and reads `"ref"`:
```kt
String branch = properties.get(CatalogProperties.REF);
```
So the configured branch is never picked up and the catalog defaults to `main`.

### Steps to reproduce:

1. Set up a Nessie destination with `main_branch_name = "staging"`.
2. Make sure `staging` exists in Nessie.
3. Run a sync.
4. Check Nessie and sync logs — all commits use `ref=main`.

### Expected behavior:

Commits should go to the configured `main_branch_name`.

### What worked for us:

We patched the connector locally by changing the property key to the literal string `"ref"`:
```
"ref" to config.icebergCatalogConfiguration.mainBranchName
```

We also updated `S3DataLakeUtilTest.kt` to assert `catalogProperties["ref"]` instead of `catalogProperties["nessie.ref"]`. After deploying the patched image, our syncs started committing to `staging` as expected.

We are not sure if this is the intended fix upstream, but it solved the issue in our environment.

### Relevant log output

```shell
Observed in Airbyte sync logs — every commit goes to Branch{name=main} regardless of the configured main_branch_name:

INFO NessieIcebergClient(commitContent):698 Committed '.' against 'Branch{name=main, metadata=null, hash=}', expected commit-id was ''

This pattern repeats for every table in the sync. The configured branch name is never applied.
```

### Contribute

- [x] Yes, I want to contribute

---
**Internal Tracking:** airbytehq/oncall#13151

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.