airbytehq / airbytehq/airbyte

Evaluate data loading approach for destination-motherduck: PyArrow vs DuckDB native read_json_auto

未關閉
#72,439 2 則留言 0 個 reaction 已指派 1 人 已被 @aaronsteers 認領 在 GitHub 檢視
area/connectors autoteam connectors/destination/motherduck team/extensibility
主要語言
Python
星號
22.1k
分支
5.3k
PR 合併指標
PR 指標待擷取

描述

## Summary

Evaluate the architectural approach for loading data into DuckDB/MotherDuck in the `destination-motherduck` connector, comparing it with PyAirbyte's approach.

> **Note for OSS Oncall Bot**: Please do not replicate or escalate this issue to the oncall repo. This is an architectural evaluation, not a bug to be fixed.

## Context

PR #70999 fixes an issue where empty Python dicts `{}` cause failures when syncing to MotherDuck. The root cause is that PyArrow infers empty dicts as "empty structs" (`struct<>`), which DuckDB rejects.

The fix in PR #70999 pre-serializes object-type columns to JSON strings using `orjson` before passing to PyArrow, which works but involves a serialize-then-parse round trip.

## Two Approaches

### Current approach (destination-motherduck)

1. Records are accumulated in Python dicts in memory (`buffer[stream_name]`)
2. PyArrow's `pa.Table.from_pydict()` converts the buffer to a PyArrow table
3. The PyArrow table is inserted into DuckDB

This approach encounters the empty struct limitation in PyArrow.

### PyAirbyte's approach

1. Records are written to JSONL files via `JsonlWriter`
2. DuckDB's native `read_json_auto()` function reads directly from those files with explicit column type mapping
3. SQL: `INSERT INTO table SELECT ... FROM read_json_auto([files], columns = {col: "JSON", ...})`

This bypasses PyArrow entirely and leverages DuckDB's native JSON handling.

## Evaluation Questions

1. **Performance**: How do the two approaches compare for various batch sizes and data shapes?
2. **Memory**: In-memory buffering vs file-based staging tradeoffs
3. **Complexity**: Is the refactor to file-based staging worth the architectural change?
4. **Consistency**: Should destination-motherduck align with PyAirbyte's approach for maintainability?

## Related

- PR #70999: Fix for empty STRUCTs (current workaround)
- PyAirbyte DuckDB processor: `airbyte/_processors/sql/duckdb.py`

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。