gendx / gendx/paralight

Support inputs with more than `u32::MAX` elements

Open
#2 0 comments 0 reactions 0 assignees View on GitHub
roadmap
Dominant language
Rust
Stars
49
Forks
2
PR merge metrics
No merged PRs in 30d

Description

Inputs with more than `u32::MAX` elements are currently not supported, because the [underlying implementation](https://github.com/gendx/paralight/blob/0.0.6/src/core/range.rs#L398) packs a `start..end` range of items as a `(u32, u32)` pair into an [`AtomicU64`](https://doc.rust-lang.org/std/sync/atomic/struct.AtomicU64.html). Being able to use an atomic type is essential for the algorithm to work, but the Rust standard library [doesn't offer](https://doc.rust-lang.org/std/sync/atomic/index.html) atomic types larger than `u64`, even though common CPUs support 128-bit atomics (e.g. [`cmpxchg16b`](https://doc.rust-lang.org/stable/core/arch/x86_64/fn.cmpxchg16b.html) on x86-64).

Possible ways forward:
- Detect CPU support and allow larger inputs on platforms that offer them.
- Depending on a crate such as [`portable_atomic`](https://docs.rs/portable-atomic/) to provide `AtomicU128` for us.
- If the input is larger than 2^32, process it into fewer than 2^32 chunks of 2^k items each, with k as small as possible.

Contributor guide

Open the contributing guide

Research direction

Start in src/core/range.rs around line 398 and inspect how the start..end range is packed into AtomicU64. Compare CPU-detection, portable_atomic, and chunking approaches against the algorithm's atomic requirements. Done should mean inputs exceeding u32::MAX are supported without breaking the parallel range-processing behavior, with validation on relevant platforms.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.