scverse / scverse/rustar-aligner

Dependency discussion: portable SIMD crate (wide/pulp) vs the hand-rolled intrinsics in simd_scan.rs

Open
#205 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

dependencies
Dominant language
Rust
Stars
75
Forks
7
Avg merge
8m
Merged PRs (30d)
1

Description

Dependency discussion per CONTRIBUTING.md. Gated on the benchmark-harness issue — this should
not turn into a PR before there is a repeatable measurement.

What exists today

Hand-rolled, and deliberately minimal:

  • src/align/simd_scan.rsfind_stop, a 16-byte chunked "does this chunk contain a stop"
    reduction in SSE2 / NEON intrinsics, with a scalar re-scan to locate the exact position. The
    module comment states the design rationale: keep the arch-specific surface to a trivial boolean
    reduction, do position extraction in portable scalar code.
  • src/cpu.rs — runtime feature detection, the build-flavour guard, and prefetch_read
    (_mm_prefetch / prfm).

That is a good design and the bar for replacing it is correspondingly high.

Candidates

Option Version License Note
wide 1.6.0 Zlib OR Apache-2.0 OR MIT Mature, fixed-width vector types, SSE2/AVX2/NEON/WASM. No multiversioning
pulp 0.22.3 MIT Built-in multiversioning, proven by faer; native-width only, so code must handle variable-width chunks
std::simd Nightly-only; the crate pins rust-version = "1.89" stable, so it is out

The actual questions

  1. Does a portable crate buy wider vectors? Today's scan is fixed at 16 bytes, i.e. SSE2/NEON
    width. AVX2 (32 B) or AVX-512 would halve or quarter the chunk count, but only with runtime
    dispatch — which is exactly what wide does not do and pulp does.
  2. Five platforms, including Windows. CONTRIBUTING.md calls this out. Any candidate has to
    build cleanly everywhere, including the aarch64 macOS path.
  3. Results must be bit-identical across dispatch paths. A scalar fallback, an SSE2 path and an
    AVX2 path must return the same find_stop index. That needs a test exercising all compiled
    paths, not just the native one.
  4. Is find_stop even hot enough to matter? Answer with a profile before touching it.

Checklist

  • Profile find_stop's share of align wall time (blocked on the benchmark harness)
  • Try widening the existing hand-rolled scan to AVX2 with is_x86_feature_detected! first,
    since that costs zero dependencies and answers question 1 directly
  • Only if that is unsatisfying: prototype with pulp (multiversioning is the differentiator)
  • Cross-path identity test: scalar vs each vector path
  • Verify Windows + macOS aarch64 builds

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read CONTRIBUTING.md and the design comments in src/align/simd_scan.rs and src/cpu.rs. Wait for the benchmark harness, then profile find_stop, try the AVX2 widening experiment, and only prototype pulp if needed. Done means measured results, cross-path index identity tests, and verified Windows and aarch64 macOS builds.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.