scverse / scverse/rustar-aligner

--limitBAMsortRAM aborts instead of spilling: evaluate an external sort for coordinate-sorted BAM

Open
#206 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Rust
Stars
75
Forks
7
Avg merge
8m
Merged PRs (30d)
1

Description

The current behaviour

src/io/bam.rs:173-181: when --limitBAMsortRAM is set and the estimate (~400 bytes/record)
exceeds it, the sort aborts with

limitBAMsortRAM={} bytes exceeded: estimated {} bytes for {} records. Increase --limitBAMsortRAM or use --outSAMtype BAM Unsorted.

Native STAR treats the same flag as a budget, not a ceiling to fail at: it bins records to disk
and merges, so a small RAM limit means more spilling, not a failed run. So on this flag we are
currently divergent in kind, not just in numbers, and the escape hatch we offer ("use Unsorted")
discards the thing the user asked for.

Candidates

Crate Note
extsort On-disk sorting over arbitrarily sized iterators; optional rayon for the in-memory buffer
ext-sort Explicit MemoryLimitedBufferBuilder
in-tree k-way merge Spill sorted runs to tempfile (already a dependency), merge on read

The in-tree option deserves a fair hearing: tempfile is already in Cargo.toml, BySJout already
buffers to a NamedTempFile (src/io/sam.rs), so the spill machinery is not foreign to this
codebase, and a k-way merge over pre-sorted runs is a small, well-understood piece of code.

Constraints

  1. The output must not move. Sorted BAM ordering is (refID, pos) with input read order as the
    tie-break; whatever sorter is used must carry an explicit tie-break key rather than relying on
    an unspecified stability guarantee.
  2. Byte-identical regardless of the RAM limit and the worker count. The limit may change how
    much spills, never what comes out.
  3. Temp files must be cleaned up on abort, and must respect --outTmpDir if that is honoured.

Checklist

  • Decide the semantics: match STAR (spill) rather than abort
  • Measure peak RSS of a sorted-BAM run on a realistic library to size the buffer default
  • Compare a crate against an in-tree k-way merge on tempfile before adding a dependency
  • Byte-identical sorted BAM across RAM limits and thread counts
  • Update the docs for --limitBAMsortRAM once the behaviour changes

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in src/io/bam.rs:173-181 to trace the current RAM-limit abort, then read src/io/sam.rs and Cargo.toml for existing temporary-file patterns and dependencies. Compare the listed crates with an in-tree k-way merge, and validate byte-identical output across RAM limits and worker counts, cleanup on abort, and --outTmpDir handling. Done means --limitBAMsortRAM spills instead of aborting and the documentation is updated.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
bioinformatics
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.