apache / apache/arrow

[C++] Add JSON/NDJSON writer to Arrow C++ core — then expose via PyArrow/Java

Open
#47,173 2 comments 0 reactions 0 assignees View on GitHub
Component: C++ Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

Arrow currently supports JSON reading in C++, Python, Java, etc. But it currently lacks any equivalent of a JSON writer. While Rust (arrow_json::writer) and Go (arrjson) implement their own serialization, they do not leverage the shared C++ core.

This results in these limitations:

- Python users (e.g., BigQuery→Arrow→Postgres JSONB, which is my particular use case) must fall back to slow, Python-level loops or fallback to orjson, missing C++‑level performance.
- No feature parity with Rust and Go, which already provide fast JSON serialization.
- Large-scale pipelines suffer from marshalling overhead and poor scaling.

## Proposal Overview

### 1. **C++ Core: Add JSON Writer API**
- Mirror the existing `arrow::json::TableReader` with a new `arrow::json::TableWriter` or `RecordBatchWriter`.
- Support both output formats:
- **NDJSON** (newline-delimited)
- **JSON array**
- Configurable via builder-pattern options:
- Include or omit nulls
- Binary types encoding (e.g., Base64)
- Formatting (pretty, flat)

### 2. **Bindings**
- **PyArrow**: add `pyarrow.json.write_json(table_or_batch, sink=None, ndjson=False, include_nulls=True, binary_encoding="base64")`, wrapping the new C++ API.
- **Arrow Java**: introduce a corresponding `JsonWriter` class to maintain cross-language feature consistency.

### 3. **Functionality & Performance**
- Full support for Arrow types: scalars, nested structs/lists, binary, timestamps, dictionaries, nulls.
- Streaming output row-by-row to avoid in-memory buffering.
- Benchmark target: achieve near-native performance, comparable to Rust’s `LineDelimitedWriter`.

*I had this request edited by an LLM, as I'm not very familiar with the Arrow backend architecture. I checked all the claims, but some inaccuracies might has slipped through, if so, sorry.

### Component(s)

C++

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.