parquet-concat - handle large number of files
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
If some process creates a large number of parquet files, it would be nice to be able to run `parquet-concat` on these (location: `parquet/src/bin/parquet-concat.rs`). The current script will however open up too many file descriptors on most systems in a situation of many files.
**Describe the solution you'd like**
Make the implementation streaming based: files are opened, read and written from in order. The current implementation starts by opening all file descriptors.
**Describe alternatives you've considered**
**Additional context**
Contributor guide
Research direction
Start at parquet/src/bin/parquet-concat.rs and inspect how the command currently opens all input files before writing. Make processing streaming-based so files are opened, read, and written in order without retaining too many file descriptors; validate it with a large number of parquet files.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- cli
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100