scverse / scverse/rustar-aligner

Align pipeline stops scaling past ~4-8 threads: decode and write are each a single thread

Open
#223 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Rust
Stars
75
Forks
7
Avg merge
8m
Merged PRs (30d)
1

Description

Measurement

Synthetic 20 Mb genome, 2 M single-end 100 bp reads, --outSAMtype BAM Unsorted, macOS aarch64,
release build (with the batch-size fix from #222 already in, so this is the remaining ceiling):

--runThreadN wall user CPU speedup
1 8.81s 11.67s 1.00x
2 4.82s 12.36s 1.83x
4 2.80s 13.49s 3.15x
8 2.43s 15.79s 3.63x
12 2.44s 17.19s 3.61x

Throughput stops improving after ~8 threads while user CPU keeps climbing and sys roughly doubles
(1.96s → 4.45s). On a 32- or 64-core machine the extra cores are being paid for and not used.

Where it goes

Same input, 8 threads, output disabled:

output wall
BAM Unsorted 2.43s
None 1.60s

So the writer stage alone is ~34% of wall at 8 threads, and it is one thread: SAM record → BAM
encode → BGZF deflate, serially, for the whole run.

Gzipped input costs on top of that (8 threads, BAM out): 2.74s from .fq.gz vs 2.43s from plain
FASTQ, i.e. ~13%, again on a single decode thread.

Why

The align pipelines are decode(1 thread) → align(N rayon workers) → write(1 thread), joined by
bounded channels. The two ends are serial, so Amdahl caps the whole thing regardless of
--runThreadN. The profile agrees: semaphore_wait_trap + __psynch_cvwait were ~16% of all
samples, i.e. workers parked waiting on the ends.

Options, in payoff order

  1. Multithreaded BGZF writer. noodles-bgzf 0.50+ runs its multithreaded writer on its own
    local pool with a supported with_worker_count, so it no longer competes with rayon's global
    pool. Biggest single lever, and the version bump is already in #207 / #211. Needs care: BGZF
    block boundaries must stay byte-identical, or the change must be documented.
  2. Parallel or chunked decode. Related to #95 (paraseq, built for exactly this) and #97
    (parallel readers). For .gz specifically, note that a BGZF-compressed FASTQ can be inflated in
    parallel with the noodles-bgzf we already depend on, with no new dependency.
  3. Only then consider reshaping the stage handoff itself (work-stealing over batches rather
    than fixed producer/consumer threads). Do not start here: it is the largest change and the
    smallest measured share.

Note

The numbers above are from a synthetic genome on a laptop; the shape (serial ends capping the
scaling) is architectural, but the exact crossover point should be re-measured on a real genome and
a real machine before sizing any of the fixes. That needs the benchmark harness from the
dependency sweep.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the benchmark harness from the dependency sweep and reproduce the decode(1 thread) → align(N workers) → write(1 thread) measurements on a real genome and machine. Read the current pipeline ends and the noodles-bgzf version changes referenced in #207 and #211, along with related issues #95 and #97. Done should include a measured improvement in scaling beyond roughly 4–8 threads without regressing output behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.