Upgrade `rand` to `0.10`, `rand_distr` to `0.6`
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
Dependabot has a PR to upgrade `rand_distr`:
- https://github.com/apache/datafusion/pull/20403
This is currently blocked as it requires `rand = "0.10"`:
- https://github.com/rust-random/rand_distr/blob/66f6791fd846ff28af51ac55d16032605a048256/CHANGELOG.md#060--2026-02-10
However it seems dependabot isn't raising a PR for `rand = "0.10"` because it removed a feature we use, `small_rng`.
### Describe the solution you'd like
Manually upgrade to `rand = "0.10"` and `rand_dist = "0.6"`
https://github.com/apache/datafusion/blob/89e14f1de500e3f117574d91443b2c22520105c6/datafusion/core/Cargo.toml#L175-L176
https://github.com/apache/datafusion/blob/89e14f1de500e3f117574d91443b2c22520105c6/Cargo.toml#L188
What is involved:
- Remove use of `small_rng` feature
- Minimal impact (code we rely on wasn't removed, just the feature gate)
- Reference: https://github.com/rust-random/rand/pull/1732
- Replace imports of `rand::Rng` with `rand::RngExt`
- Reference: https://github.com/rust-random/rand/pull/1717
- `StdRng` is no longer `Clone`, fix that somehow:
- Reference: https://github.com/rust-random/rand/pull/1677
https://github.com/apache/datafusion/blob/89e14f1de500e3f117574d91443b2c22520105c6/datafusion/common/benches/with_hashes.rs#L238
- `Fill` no longer implemented for slices
- Reference: https://github.com/rust-random/rand/pull/1652
https://github.com/apache/datafusion/blob/89e14f1de500e3f117574d91443b2c22520105c6/datafusion/functions/src/math/random.rs#L85
- `arrow-rs` still depends on `rand = "0.9"` and we interface with this, so we need to upgrade `arrow-rs`
- `create_primitive_array` comes from `arrow-rs`
https://github.com/apache/datafusion/blob/89e14f1de500e3f117574d91443b2c22520105c6/datafusion/functions-aggregate/benches/array_agg.rs#L91
### Describe alternatives you've considered
_No response_
### Additional context
See changelog for rand:
- https://github.com/rust-random/rand/blob/b1ad27346a9637979c2e286988b0aa55f771ecc9/CHANGELOG.md#0100---2026-02-08
Contributor guide
Research direction
Start with datafusion/core/Cargo.toml and the workspace Cargo.toml entries for rand and rand_distr, then inspect the referenced uses in datafusion/common/benches/with_hashes.rs, datafusion/functions/src/math/random.rs, and datafusion/functions-aggregate/benches/array_agg.rs. Update the dependencies and resolve the listed rand API changes, including the arrow-rs compatibility issue. Done means the project builds with rand 0.10 and rand_distr 0.6 without the small_rng feature.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, build-system
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100