apache / apache/parquet-java

Why doesn't Parquet currently support writing multiple row groups simultaneously?

Open
#2,929 1 comment 0 reactions 0 assignees View on GitHub
question
Dominant language
Java
Stars
3.1k
Forks
1.6k
Avg merge
3d 12h
Merged PRs (30d)
33

Description

Hi Parquet developers,

I have a question regarding the current implementation of Parquet. As far as I understand, Parquet does not support writing multiple row groups simultaneously. Could you please explain the reasoning behind this design choice?

Additionally, I am considering modifying Parquet to allow for multiple row groups to exist in memory and be flushed sequentially. From a high-level perspective, does this approach seem feasible? Are there any potential pitfalls or challenges I should be aware of?

Thank you for your time and assistance.

Best regards,

Contributor guide

No contributing guide indexed for this repository

Research direction

No source file, test, or entry point is named. Start by locating the current Parquet writer and row-group handling, then trace how row groups are held and flushed; done would require a maintainer-confirmed design and documented answers about feasibility, memory use, ordering, and other pitfalls.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.