apache / apache/arrow-rs

parquet-concat - handle large number of files

Open
#8,650 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

If some process creates a large number of parquet files, it would be nice to be able to run `parquet-concat` on these (location: `parquet/src/bin/parquet-concat.rs`). The current script will however open up too many file descriptors on most systems in a situation of many files.

**Describe the solution you'd like**

Make the implementation streaming based: files are opened, read and written from in order. The current implementation starts by opening all file descriptors.

**Describe alternatives you've considered**

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start at parquet/src/bin/parquet-concat.rs and inspect how the command currently opens all input files before writing. Make processing streaming-based so files are opened, read, and written in order without retaining too many file descriptors; validate it with a large number of parquet files.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
cli
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.