Why no bulk Arrow→Parquet write API in Java? How to avoid row-by-row RecordConsumer + optimize?
- 主要语言
- Java
- 星标
- 3.1k
- 派生
- 1.6k
- 平均合并
- 3 天 12 小时
- 30 天内合并 PR
- 33
描述
### Describe the usage question you have. Please include as many useful details as possible.
I have ~ 20KB objects that I need to write to Parquet efficiently from Java.
In C++, C#, and Python there's a direct/bulk Arrow-Parquet write (e.g. WriteTable / write_table) that avoids row-by-row iteration, but in Java I only see row-by-row paths via RecordConsumer or internal/unstable column writers.
Questions:
1. Is there a supported bulk/columnar Arrow-Parquet write API in Java (e.g, VectorSchemaRoot
→ Parquet) that avoids row-by-row calls?
2. If not, why is Java limited to row-by-row writes today? Any roadmap for feature parity with C++/Python/C#?
3. For now, what's the recommended optimization path to write 20KB objects at high throughput from Java (without JNI), or is JNI/Dataset the recommended route?
4. Any best practices (batch sizing, encodings, writer settings) to mitigate the row-by-row overhead?
Thanks
### Component(s)
_No response_
贡献指南
这个仓库没有索引到贡献指南
调研方向
首先检查 issue 中提到的 Java 写入路径,尤其是 RecordConsumer、内部列写入器和 VectorSchemaRoot,然后将它们与针对 C++、C# 和 Python 描述的 bulk API 进行比较。完成的标准是记录是否存在受支持的 bulk Java 路径,说明当前的限制或 roadmap,并确定不使用 JNI 的推荐优化路径。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- java
- 领域
- data-engineering
- Issue 类型
- 功能
- 难度
- 5/5
- 预计耗时
- 一周以上
- 活跃度
- 停滞
- 描述清晰度
- 需要澄清
- 新手友好度
- 25/100