airbytehq / airbytehq/airbyte

Evaluate data loading approach for destination-motherduck: PyArrow vs DuckDB native read_json_auto

Aberta
#72,439 2 comentários 0 reações 1 responsável Reivindicada por @aaronsteers Ver no GitHub
area/connectors autoteam connectors/destination/motherduck team/extensibility
Linguagem predominante
Python
Estrelas
22.1k
Forks
5.4k
Merge médio
5h
PRs com merge (30d)
671

Descrição

## Summary

Evaluate the architectural approach for loading data into DuckDB/MotherDuck in the `destination-motherduck` connector, comparing it with PyAirbyte's approach.

> **Note for OSS Oncall Bot**: Please do not replicate or escalate this issue to the oncall repo. This is an architectural evaluation, not a bug to be fixed.

## Context

PR #70999 fixes an issue where empty Python dicts `{}` cause failures when syncing to MotherDuck. The root cause is that PyArrow infers empty dicts as "empty structs" (`struct<>`), which DuckDB rejects.

The fix in PR #70999 pre-serializes object-type columns to JSON strings using `orjson` before passing to PyArrow, which works but involves a serialize-then-parse round trip.

## Two Approaches

### Current approach (destination-motherduck)

1. Records are accumulated in Python dicts in memory (`buffer[stream_name]`)
2. PyArrow's `pa.Table.from_pydict()` converts the buffer to a PyArrow table
3. The PyArrow table is inserted into DuckDB

This approach encounters the empty struct limitation in PyArrow.

### PyAirbyte's approach

1. Records are written to JSONL files via `JsonlWriter`
2. DuckDB's native `read_json_auto()` function reads directly from those files with explicit column type mapping
3. SQL: `INSERT INTO table SELECT ... FROM read_json_auto([files], columns = {col: "JSON", ...})`

This bypasses PyArrow entirely and leverages DuckDB's native JSON handling.

## Evaluation Questions

1. **Performance**: How do the two approaches compare for various batch sizes and data shapes?
2. **Memory**: In-memory buffering vs file-based staging tradeoffs
3. **Complexity**: Is the refactor to file-based staging worth the architectural change?
4. **Consistency**: Should destination-motherduck align with PyAirbyte's approach for maintainability?

## Related

- PR #70999: Fix for empty STRUCTs (current workaround)
- PyAirbyte DuckDB processor: `airbyte/_processors/sql/duckdb.py`

Guia de contribuição

Abrir o guia de contribuição

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.