Evaluate column name collision handling: PyAirbyte vs MotherDuck approaches
- 主要言語
- Python
- スター
- 22.1k
- フォーク
- 5.3k
- PR マージ指標
- PR 指標を取得中
説明
## Summary
This issue tracks the evaluation of two different approaches for handling column name collisions after normalization in Airbyte destinations.
**Context:** [PR #70438](https://github.com/airbytehq/airbyte/pull/70438) fixes a bug in the MotherDuck destination where camelCase columns were always NULL. However, it also introduces a new behavior for handling duplicate keys after normalization that differs from PyAirbyte's approach.
## The Problem
When source data contains fields that normalize to the same column name (e.g., `UserID` and `userid` both normalize to `userid`), destinations must decide how to handle this collision.
## Current Approaches
### PyAirbyte (in `airbyte/records.py`)
```python
self.update({self._stream_handler.to_index_case(k): v for k, v in from_dict.items()})
```
- **Behavior:** Last value wins (standard dict update)
- **Data loss:** One field value is silently overwritten
- **User notification:** None
### MotherDuck PR #70438
```python
if len(normalized_keys) < len(data):
logger.warning("Data contained duplicate keys after normalization...")
continue # Skip entire record
```
- **Behavior:** Skip the entire record
- **Data loss:** Entire record is dropped
- **User notification:** Warning logged
## Comparison
| Scenario | PyAirbyte | MotherDuck PR |
|----------|-----------|---------------|
| `{"UserID": 1, "userid": 2}` | Keeps `userid: 2` (last wins) | Skips entire record |
| Data loss | One field value | Entire record |
| User notification | None | Warning logged |
## Questions to Evaluate
1. Should there be a consistent approach across all Airbyte destinations?
2. Is "last wins" more forgiving (preserves most data) or more dangerous (silent data loss)?
3. Is "skip record" more conservative (explicit about data issues) or more harmful (loses entire records)?
4. Should this be configurable by the user?
5. Should we add warnings to PyAirbyte's approach as well?
## Related
- PR: https://github.com/airbytehq/airbyte/pull/70438
- PyAirbyte records handling: https://github.com/airbytehq/PyAirbyte/blob/main/airbyte/records.py
---
**Note to OSS Oncall bot:** Please do NOT replicate or escalate this issue to the oncall repo. This is a design discussion, not a bug to be fixed.
コントリビューションガイド
評価
この issue はまだ評価されていません。