scverse / scverse/rustar-aligner
Dependency discussion: optional rapidgzip feature for parallel gzip input
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 75
- Forks
- 7
- Avg merge
- 8m
- Merged PRs (30d)
- 1
Description
Raising this before the PR, as CONTRIBUTING.md requires for a new dependency, especially a
non-Rust one.
Proposal
Add rapidgzip 1.2.1 (MIT OR Apache-2.0, released
2026-04-01) as an optional cargo feature, off by default, for parallel decoding of gzipped
--readFilesIn. It is a safe Rust wrapper over rapidgzip-sys, which vendors ~20 MB of C++
(rapidgzip + zlib + rpmalloc) and builds it with CMake.
What it costs
- A CMake and C++17 toolchain requirement for anyone building with the feature on. This is why
it must be a feature and not a default dependency: the crate is published to crates.io and builds
on five platforms including Windows. - A second allocator (rpmalloc) inside a process that installs mimalloc globally.
- A young crate: first release 2026-03-25, ~2.3k downloads.
What it buys
Standalone, decoding a 73 MB level-6 .fq.gz (427 MB out) on 16 logical cores: flate2+zlib-rs
1300 MB/s on one thread, rapidgzip 548 MB/s on one, 1524 MB/s on four, 1705 MB/s on eight. Full
tables in #223.
End to end inside the aligner, 20 Mb genome, 2 M reads, --runThreadN 8, median of three:
| decode threads | wall | user CPU | peak RSS |
|---|---|---|---|
0 (flate2) |
3.18s | 25.3s | 1242 MB |
| 2 | 3.21s | 25.7s | 2069 MB |
| 4 | 3.33s | 26.2s | 2100 MB |
No wall-clock gain on this workload, and ~800 MB more resident for the decoder's chunk cache.
That is consistent with the profile: decode already runs on its own thread overlapped with
alignment, gzipped input costs only ~11% of wall, and the serial writer stage is ~34% (#223).
So why add it at all
For the input-bound shape this does not cover: fast alignment, slow single-stream decode, spare
cores. And to make that measurable rather than hypothetical. It stays off by default and inert
until RUSTAR_GZ_DECODE_THREADS is set, so nobody pays for it accidentally.
The zero-dependency alternative that already exists: --readFilesCommand "rapidgzip -dc" (or
igzip -dc, pigz -dc).
Question for the maintainer
Is an optional feature with a C++/CMake build path acceptable at all here, or would you rather this
stayed outside the tree and be handled through --readFilesCommand? PR is ready either way; say
the word and I will close it.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with CONTRIBUTING.md and the benchmark tables referenced in #223, then compare the proposed optional rapidgzip path with the existing --readFilesCommand alternative. Done means the maintainer decides whether the C++/CMake feature belongs in the tree; the proposal is ready for either outcome.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cmake, cpp, rust
- Domain
- build-system, cli, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100