airbytehq / airbytehq/airbyte

Evaluate column name collision handling: PyAirbyte vs MotherDuck approaches

オープン
#72,440 コメント 2 件 リアクション 0 件 担当者 1 名 @aaronsteers が担当を希望しています GitHub で見る
team/triage
主要言語
Python
スター
22.1k
フォーク
5.3k
PR マージ指標
PR 指標を取得中

説明

## Summary

This issue tracks the evaluation of two different approaches for handling column name collisions after normalization in Airbyte destinations.

**Context:** [PR #70438](https://github.com/airbytehq/airbyte/pull/70438) fixes a bug in the MotherDuck destination where camelCase columns were always NULL. However, it also introduces a new behavior for handling duplicate keys after normalization that differs from PyAirbyte's approach.

## The Problem

When source data contains fields that normalize to the same column name (e.g., `UserID` and `userid` both normalize to `userid`), destinations must decide how to handle this collision.

## Current Approaches

### PyAirbyte (in `airbyte/records.py`)

```python
self.update({self._stream_handler.to_index_case(k): v for k, v in from_dict.items()})
```

- **Behavior:** Last value wins (standard dict update)
- **Data loss:** One field value is silently overwritten
- **User notification:** None

### MotherDuck PR #70438

```python
if len(normalized_keys) < len(data):
logger.warning("Data contained duplicate keys after normalization...")
continue # Skip entire record
```

- **Behavior:** Skip the entire record
- **Data loss:** Entire record is dropped
- **User notification:** Warning logged

## Comparison

| Scenario | PyAirbyte | MotherDuck PR |
|----------|-----------|---------------|
| `{"UserID": 1, "userid": 2}` | Keeps `userid: 2` (last wins) | Skips entire record |
| Data loss | One field value | Entire record |
| User notification | None | Warning logged |

## Questions to Evaluate

1. Should there be a consistent approach across all Airbyte destinations?
2. Is "last wins" more forgiving (preserves most data) or more dangerous (silent data loss)?
3. Is "skip record" more conservative (explicit about data issues) or more harmful (loses entire records)?
4. Should this be configurable by the user?
5. Should we add warnings to PyAirbyte's approach as well?

## Related

- PR: https://github.com/airbytehq/airbyte/pull/70438
- PyAirbyte records handling: https://github.com/airbytehq/PyAirbyte/blob/main/airbyte/records.py

---

**Note to OSS Oncall bot:** Please do NOT replicate or escalate this issue to the oncall repo. This is a design discussion, not a bug to be fixed.

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。