airbytehq / airbytehq/airbyte

[destination-s3-data-lake] main_branch_name is ignored for Nessie; all commits go to main

Aberta Para iniciantes
#82,732 2 comentários 0 reações 0 responsáveis Ver no GitHub
area/connectors autoteam community connectors/destination/s3-data-lake needs-triage team/use type/bug
Linguagem predominante
Python
Estrelas
22.1k
Forks
5.4k
Merge médio
5h
PRs com merge (30d)
671

Descrição

### Connector Name

destination-s3-data-lake

### Connector Version

0.3.52

### What step the error happened?

During the sync

### Relevant information

We are running Airbyte 2.0.1 with the `destination-s3-data-lake` connector against a Nessie catalog. We configured `main_branch_name: staging` and confirmed the config is passed through to the connector pod. The `staging` branch exists in Nessie. Still, every sync and destination test commits to `main`.

We traced this to `S3DataLakeUtil.kt`. The connector currently sets:
```
NessieConfigConstants.CONF_NESSIE_REF to config.icebergCatalogConfiguration.mainBranchName
```

This produces the property key `"nessie.ref"`. But Iceberg's NessieCatalog.initialize() strips the `"nessie."` prefix and reads `"ref"`:
```kt
String branch = properties.get(CatalogProperties.REF);
```
So the configured branch is never picked up and the catalog defaults to `main`.

### Steps to reproduce:

1. Set up a Nessie destination with `main_branch_name = "staging"`.
2. Make sure `staging` exists in Nessie.
3. Run a sync.
4. Check Nessie and sync logs — all commits use `ref=main`.

### Expected behavior:

Commits should go to the configured `main_branch_name`.

### What worked for us:

We patched the connector locally by changing the property key to the literal string `"ref"`:
```
"ref" to config.icebergCatalogConfiguration.mainBranchName
```

We also updated `S3DataLakeUtilTest.kt` to assert `catalogProperties["ref"]` instead of `catalogProperties["nessie.ref"]`. After deploying the patched image, our syncs started committing to `staging` as expected.

We are not sure if this is the intended fix upstream, but it solved the issue in our environment.

### Relevant log output

```shell
Observed in Airbyte sync logs — every commit goes to Branch{name=main} regardless of the configured main_branch_name:

INFO NessieIcebergClient(commitContent):698 Committed '.' against 'Branch{name=main, metadata=null, hash=}', expected commit-id was ''

This pattern repeats for every table in the sync. The configured branch name is never applied.
```

### Contribute

- [x] Yes, I want to contribute

---
**Internal Tracking:** airbytehq/oncall#13151

Guia de contribuição

Abrir o guia de contribuição

Direção de pesquisa

Start in S3DataLakeUtil.kt and inspect how the Nessie branch configuration is mapped into catalog properties. Run S3DataLakeUtilTest.kt, update its assertions to match the property consumed by NessieCatalog, and verify that configuring main_branch_name causes commits to use that branch instead of main.

Escrita pelo modelo de indexação a partir do texto da issue.

Avaliação

Stack de tecnologia
kotlin
Domínio
data-engineering
Tipo de issue
Bug
Dificuldade
2/5
Tempo estimado
1-3 horas
Status de atividade
Pouca atividade
Clareza
Claramente especificada
Facilidade para iniciantes
78/100

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.