apache / apache/arrow-rs

Blog post about changes to write wide columns, large string columns

Open
#10,073 3 comments 4 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

@adriangb has made some great contributions to the Rust parquet writer to support several usecases where parquet writers typically struggle:
- https://github.com/apache/arrow-rs/pull/9972
-https://github.com/apache/arrow-rs/pull/10020

I think it would be amazing to write a blog post explaining the challenge and the solutions

Challenge 1: Oversized pages for large binary/string columns
- https://github.com/apache/arrow-rs/issues/10061

Challenge 2: RAM buffering requirements when writing large Row Groups
- https://github.com/apache/arrow-rs/issues/10071

The idea is to write a blog in https://arrow.apache.org/blog/ (source in https://github.com/apache/arrow-site) that explains the challenges and how we solved it with Software Engineering rather than a new file format

Contributor guide

Open the contributing guide

Research direction

Start by reading the linked arrow-rs pull requests and issues 10061 and 10071 to understand the two parquet writer challenges and their solutions. Then inspect the blog source in the apache/arrow-site repository. Done means a published post on arrow.apache.org/blog explaining the challenges and how software engineering addressed them without introducing a new file format.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
documentation
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.