Why doesn't Parquet currently support writing multiple row groups simultaneously?
- Dominant language
- Java
- Stars
- 3.1k
- Forks
- 1.6k
- Avg merge
- 3d 12h
- Merged PRs (30d)
- 33
Description
Hi Parquet developers,
I have a question regarding the current implementation of Parquet. As far as I understand, Parquet does not support writing multiple row groups simultaneously. Could you please explain the reasoning behind this design choice?
Additionally, I am considering modifying Parquet to allow for multiple row groups to exist in memory and be flushed sequentially. From a high-level perspective, does this approach seem feasible? Are there any potential pitfalls or challenges I should be aware of?
Thank you for your time and assistance.
Best regards,
Contributor guide
No contributing guide indexed for this repository
Research direction
No source file, test, or entry point is named. Start by locating the current Parquet writer and row-group handling, then trace how row groups are held and flushed; done would require a maintainer-confirmed design and documented answers about feasibility, memory use, ordering, and other pitfalls.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100