scverse / scverse/rustar-aligner

Dependency discussion: libsais (vendored C via cc) for suffix-array construction

Open
#162 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

dependencies
Dominant language
Rust
Stars
75
Forks
7
Avg merge
8m
Merged PRs (30d)
1

Description

CONTRIBUTING.md asks for a new dependency, especially a non-Rust one, to be raised in an issue before the PR. #109 predates that rule and adds one, so here are the facts and the trade-offs. The decision is yours; I am not arguing for it here, only laying it out.

What it is

Crate libsais 0.2.0, wrapping libsais-sys 0.2.0
Upstream https://github.com/feldroop/libsais-rs, wrapping IlyaGrebnov/libsais
Licence MIT OR Apache-2.0, both crates
New transitive deps bytemuck, either, and cc as a build dependency
Declared as libsais = { version = "0.2.0", default-features = false }

What it does to the build

The C is vendored in libsais-sys (libsais/src/*.c, libsais/include/*.h) and compiled by cc. There is no system library to install, no bindgen, no libclang, and no pkg-config probe. cc pulls in find-msvc-tools on Windows, which is how it locates MSVC.

default-features = false matters: the crate's default feature is openmp, which would add an OpenMP toolchain requirement. It is off, so the C is built single-threaded and nothing outside the vendored sources is needed.

The five-platform CI matrix on #109 is green, windows-x86_64 included.

What it buys

Suffix-array construction time, on this branch, --runThreadN 8, same machine, warm cache. Both builders produce byte-identical SA and SAindex, because the suffix array of a text is unique.

genome size libsais caps-sa ratio
yeast R64-1-1 12 Mb 1.0 s / 0.39 GB 1.7 s / 0.22 GB 1.6x
human chr1 249 Mb 27.4 s / 6.5 GB 47.2 s / 5.6 GB 1.7x
GRCh38 primary 3.1 Gb 641.6 s / 86.8 GB 1040.4 s / 14.4 GB 1.6x

--runThreadN 8, same machine, warm cache. The ratio holds across three orders of magnitude. libsais is single-threaded here, because default-features = false turns off its OpenMP feature; it still wins because caps-sa spends about six times the total CPU work for the same array (on GRCh38: 726 s of user time against 6135 s).

What it costs

  • A C compiler in the build. Today the crate graph is pure Rust; this changes that for everyone who builds from source, including crates.io consumers.
  • Supply chain. Two more crates and a vendored C library to track for advisories. cargo audit is clean today.
  • Memory. libsais is in-memory. Its peak is roughly 17 × 2N bytes, so a mammalian genome needs on the order of 100 GB where the external-memory builder needs a few. #109 gates the choice on --limitGenomeGenerateRAM for exactly that reason, and at the 31 GB default a human genome takes the caps-sa path, not this one.

That last point is the honest summary of the scope: the win lands on genomes up to roughly chromosome scale, which is where most of the test and development loop sits, and the default declines it above that.

Alternatives

  • Do nothing. caps-sa already builds correct indices, and genomeGenerate on GRCh38 is already faster than STAR 2.7.11b on this branch's predecessor work (7:36 against 11:26 wall, 32 threads). The gain here is on smaller genomes.
  • Pure-Rust SA-IS in tree. No new dependency and no C, at the cost of writing and maintaining a construction algorithm that libsais has spent years tuning. It would also have to be re-validated for byte-identity.
  • Optional feature. Put libsais behind a non-default Cargo feature so the default build stays pure Rust and packagers opt in. This costs a CI dimension and means the fast path is not what most users get.

If the answer is no, #109 closes and nothing is lost but the branch. If it is "yes, but behind a feature", say so and I will restructure it that way.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with CONTRIBUTING.md and the dependency changes described in #109. Review the reported CI matrix, libsais memory trade-offs, and benchmark results before deciding whether the dependency should be accepted, made optional, or rejected. Done means the dependency policy decision is recorded and #109 is either restructured accordingly or closed.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, rust
Domain
bioinformatics, build-system, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.