[destination-s3-data-lake] main_branch_name is ignored for Nessie; all commits go to main
- Linguagem predominante
- Python
- Estrelas
- 22.1k
- Forks
- 5.4k
- Merge médio
- 5h
- PRs com merge (30d)
- 671
Descrição
### Connector Name
destination-s3-data-lake
### Connector Version
0.3.52
### What step the error happened?
During the sync
### Relevant information
We are running Airbyte 2.0.1 with the `destination-s3-data-lake` connector against a Nessie catalog. We configured `main_branch_name: staging` and confirmed the config is passed through to the connector pod. The `staging` branch exists in Nessie. Still, every sync and destination test commits to `main`.
We traced this to `S3DataLakeUtil.kt`. The connector currently sets:
```
NessieConfigConstants.CONF_NESSIE_REF to config.icebergCatalogConfiguration.mainBranchName
```
This produces the property key `"nessie.ref"`. But Iceberg's NessieCatalog.initialize() strips the `"nessie."` prefix and reads `"ref"`:
```kt
String branch = properties.get(CatalogProperties.REF);
```
So the configured branch is never picked up and the catalog defaults to `main`.
### Steps to reproduce:
1. Set up a Nessie destination with `main_branch_name = "staging"`.
2. Make sure `staging` exists in Nessie.
3. Run a sync.
4. Check Nessie and sync logs — all commits use `ref=main`.
### Expected behavior:
Commits should go to the configured `main_branch_name`.
### What worked for us:
We patched the connector locally by changing the property key to the literal string `"ref"`:
```
"ref" to config.icebergCatalogConfiguration.mainBranchName
```
We also updated `S3DataLakeUtilTest.kt` to assert `catalogProperties["ref"]` instead of `catalogProperties["nessie.ref"]`. After deploying the patched image, our syncs started committing to `staging` as expected.
We are not sure if this is the intended fix upstream, but it solved the issue in our environment.
### Relevant log output
```shell
Observed in Airbyte sync logs — every commit goes to Branch{name=main} regardless of the configured main_branch_name:
INFO NessieIcebergClient(commitContent):698 Committed '.' against 'Branch{name=main, metadata=null, hash=}', expected commit-id was ''
This pattern repeats for every table in the sync. The configured branch name is never applied.
```
### Contribute
- [x] Yes, I want to contribute
---
**Internal Tracking:** airbytehq/oncall#13151
Guia de contribuição
Direção de pesquisa
Start in S3DataLakeUtil.kt and inspect how the Nessie branch configuration is mapped into catalog properties. Run S3DataLakeUtilTest.kt, update its assertions to match the property consumed by NessieCatalog, and verify that configuring main_branch_name causes commits to use that branch instead of main.
Escrita pelo modelo de indexação a partir do texto da issue.
Avaliação
- Stack de tecnologia
- kotlin
- Domínio
- data-engineering
- Tipo de issue
- Bug
- Dificuldade
- 2/5
- Tempo estimado
- 1-3 horas
- Status de atividade
- Pouca atividade
- Clareza
- Claramente especificada
- Facilidade para iniciantes
- 78/100