scverse / scverse/rustar-aligner

Dependency discussion: optional rapidgzip feature for parallel gzip input

Open
#224 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

dependencies
Dominant language
Rust
Stars
75
Forks
7
Avg merge
8m
Merged PRs (30d)
1

Description

Raising this before the PR, as CONTRIBUTING.md requires for a new dependency, especially a
non-Rust one.

Proposal

Add rapidgzip 1.2.1 (MIT OR Apache-2.0, released
2026-04-01) as an optional cargo feature, off by default, for parallel decoding of gzipped
--readFilesIn. It is a safe Rust wrapper over rapidgzip-sys, which vendors ~20 MB of C++
(rapidgzip + zlib + rpmalloc) and builds it with CMake.

What it costs

  • A CMake and C++17 toolchain requirement for anyone building with the feature on. This is why
    it must be a feature and not a default dependency: the crate is published to crates.io and builds
    on five platforms including Windows.
  • A second allocator (rpmalloc) inside a process that installs mimalloc globally.
  • A young crate: first release 2026-03-25, ~2.3k downloads.

What it buys

Standalone, decoding a 73 MB level-6 .fq.gz (427 MB out) on 16 logical cores: flate2+zlib-rs
1300 MB/s on one thread, rapidgzip 548 MB/s on one, 1524 MB/s on four, 1705 MB/s on eight. Full
tables in #223.

End to end inside the aligner, 20 Mb genome, 2 M reads, --runThreadN 8, median of three:

decode threads wall user CPU peak RSS
0 (flate2) 3.18s 25.3s 1242 MB
2 3.21s 25.7s 2069 MB
4 3.33s 26.2s 2100 MB

No wall-clock gain on this workload, and ~800 MB more resident for the decoder's chunk cache.
That is consistent with the profile: decode already runs on its own thread overlapped with
alignment, gzipped input costs only ~11% of wall, and the serial writer stage is ~34% (#223).

So why add it at all

For the input-bound shape this does not cover: fast alignment, slow single-stream decode, spare
cores. And to make that measurable rather than hypothetical. It stays off by default and inert
until RUSTAR_GZ_DECODE_THREADS is set, so nobody pays for it accidentally.

The zero-dependency alternative that already exists: --readFilesCommand "rapidgzip -dc" (or
igzip -dc, pigz -dc).

Question for the maintainer

Is an optional feature with a C++/CMake build path acceptable at all here, or would you rather this
stayed outside the tree and be handled through --readFilesCommand? PR is ready either way; say
the word and I will close it.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with CONTRIBUTING.md and the benchmark tables referenced in #223, then compare the proposed optional rapidgzip path with the existing --readFilesCommand alternative. Done means the maintainer decides whether the C++/CMake feature belongs in the tree; the proposal is ready for either outcome.

Written by the indexing model from the issue text.

Assessment

Tech stack
cmake, cpp, rust
Domain
build-system, cli, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.