scverse / scverse/rustar-aligner
--limitBAMsortRAM aborts instead of spilling: evaluate an external sort for coordinate-sorted BAM
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 75
- Forks
- 7
- Avg merge
- 8m
- Merged PRs (30d)
- 1
Description
The current behaviour
src/io/bam.rs:173-181: when --limitBAMsortRAM is set and the estimate (~400 bytes/record)
exceeds it, the sort aborts with
limitBAMsortRAM={} bytes exceeded: estimated {} bytes for {} records. Increase --limitBAMsortRAM or use --outSAMtype BAM Unsorted.
Native STAR treats the same flag as a budget, not a ceiling to fail at: it bins records to disk
and merges, so a small RAM limit means more spilling, not a failed run. So on this flag we are
currently divergent in kind, not just in numbers, and the escape hatch we offer ("use Unsorted")
discards the thing the user asked for.
Candidates
| Crate | Note |
|---|---|
extsort |
On-disk sorting over arbitrarily sized iterators; optional rayon for the in-memory buffer |
ext-sort |
Explicit MemoryLimitedBufferBuilder |
| in-tree k-way merge | Spill sorted runs to tempfile (already a dependency), merge on read |
The in-tree option deserves a fair hearing: tempfile is already in Cargo.toml, BySJout already
buffers to a NamedTempFile (src/io/sam.rs), so the spill machinery is not foreign to this
codebase, and a k-way merge over pre-sorted runs is a small, well-understood piece of code.
Constraints
- The output must not move. Sorted BAM ordering is
(refID, pos)with input read order as the
tie-break; whatever sorter is used must carry an explicit tie-break key rather than relying on
an unspecified stability guarantee. - Byte-identical regardless of the RAM limit and the worker count. The limit may change how
much spills, never what comes out. - Temp files must be cleaned up on abort, and must respect
--outTmpDirif that is honoured.
Checklist
- Decide the semantics: match STAR (spill) rather than abort
- Measure peak RSS of a sorted-BAM run on a realistic library to size the buffer default
- Compare a crate against an in-tree k-way merge on
tempfilebefore adding a dependency - Byte-identical sorted BAM across RAM limits and thread counts
- Update the docs for
--limitBAMsortRAMonce the behaviour changes
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in src/io/bam.rs:173-181 to trace the current RAM-limit abort, then read src/io/sam.rs and Cargo.toml for existing temporary-file patterns and dependencies. Compare the listed crates with an in-tree k-way merge, and validate byte-identical output across RAM limits and worker counts, cleanup on abort, and --outTmpDir handling. Done means --limitBAMsortRAM spills instead of aborting and the documentation is updated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- bioinformatics
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100