Evaluate data loading approach for destination-motherduck: PyArrow vs DuckDB native read_json_auto
- 主要語言
- Python
- 星號
- 22.1k
- 分支
- 5.3k
- PR 合併指標
- PR 指標待擷取
描述
## Summary
Evaluate the architectural approach for loading data into DuckDB/MotherDuck in the `destination-motherduck` connector, comparing it with PyAirbyte's approach.
> **Note for OSS Oncall Bot**: Please do not replicate or escalate this issue to the oncall repo. This is an architectural evaluation, not a bug to be fixed.
## Context
PR #70999 fixes an issue where empty Python dicts `{}` cause failures when syncing to MotherDuck. The root cause is that PyArrow infers empty dicts as "empty structs" (`struct<>`), which DuckDB rejects.
The fix in PR #70999 pre-serializes object-type columns to JSON strings using `orjson` before passing to PyArrow, which works but involves a serialize-then-parse round trip.
## Two Approaches
### Current approach (destination-motherduck)
1. Records are accumulated in Python dicts in memory (`buffer[stream_name]`)
2. PyArrow's `pa.Table.from_pydict()` converts the buffer to a PyArrow table
3. The PyArrow table is inserted into DuckDB
This approach encounters the empty struct limitation in PyArrow.
### PyAirbyte's approach
1. Records are written to JSONL files via `JsonlWriter`
2. DuckDB's native `read_json_auto()` function reads directly from those files with explicit column type mapping
3. SQL: `INSERT INTO table SELECT ... FROM read_json_auto([files], columns = {col: "JSON", ...})`
This bypasses PyArrow entirely and leverages DuckDB's native JSON handling.
## Evaluation Questions
1. **Performance**: How do the two approaches compare for various batch sizes and data shapes?
2. **Memory**: In-memory buffering vs file-based staging tradeoffs
3. **Complexity**: Is the refactor to file-based staging worth the architectural change?
4. **Consistency**: Should destination-motherduck align with PyAirbyte's approach for maintainability?
## Related
- PR #70999: Fix for empty STRUCTs (current workaround)
- PyAirbyte DuckDB processor: `airbyte/_processors/sql/duckdb.py`
貢獻指南
評估
這個 Issue 還沒有評估資料。