apache / apache/parquet-java

Why no bulk Arrow→Parquet write API in Java? How to avoid row-by-row RecordConsumer + optimize?

未关闭
#3,353 1 条评论 1 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Java
星标
3.1k
派生
1.6k
平均合并
3 天 12 小时
30 天内合并 PR
33

描述

### Describe the usage question you have. Please include as many useful details as possible.

I have ~ 20KB objects that I need to write to Parquet efficiently from Java.
In C++, C#, and Python there's a direct/bulk Arrow-Parquet write (e.g. WriteTable / write_table) that avoids row-by-row iteration, but in Java I only see row-by-row paths via RecordConsumer or internal/unstable column writers.
Questions:
1. Is there a supported bulk/columnar Arrow-Parquet write API in Java (e.g, VectorSchemaRoot
→ Parquet) that avoids row-by-row calls?
2. If not, why is Java limited to row-by-row writes today? Any roadmap for feature parity with C++/Python/C#?
3. For now, what's the recommended optimization path to write 20KB objects at high throughput from Java (without JNI), or is JNI/Dataset the recommended route?
4. Any best practices (batch sizing, encodings, writer settings) to mitigate the row-by-row overhead?

Thanks

### Component(s)

_No response_

贡献指南

这个仓库没有索引到贡献指南

调研方向

首先检查 issue 中提到的 Java 写入路径,尤其是 RecordConsumer、内部列写入器和 VectorSchemaRoot,然后将它们与针对 C++、C# 和 Python 描述的 bulk API 进行比较。完成的标准是记录是否存在受支持的 bulk Java 路径,说明当前的限制或 roadmap,并确定不使用 JNI 的推荐优化路径。

由索引模型根据 Issue 内容生成。

评估

技术栈
java
领域
data-engineering
Issue 类型
功能
难度
5/5
预计耗时
一周以上
活跃度
停滞
描述清晰度
需要澄清
新手友好度
25/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。