apache / apache/datafusion

Upgrade `rand` to `0.10`, `rand_distr` to `0.6`

Open
#21,839 3 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

Dependabot has a PR to upgrade `rand_distr`:

- https://github.com/apache/datafusion/pull/20403

This is currently blocked as it requires `rand = "0.10"`:

- https://github.com/rust-random/rand_distr/blob/66f6791fd846ff28af51ac55d16032605a048256/CHANGELOG.md#060--2026-02-10

However it seems dependabot isn't raising a PR for `rand = "0.10"` because it removed a feature we use, `small_rng`.

### Describe the solution you'd like

Manually upgrade to `rand = "0.10"` and `rand_dist = "0.6"`

https://github.com/apache/datafusion/blob/89e14f1de500e3f117574d91443b2c22520105c6/datafusion/core/Cargo.toml#L175-L176

https://github.com/apache/datafusion/blob/89e14f1de500e3f117574d91443b2c22520105c6/Cargo.toml#L188

What is involved:

- Remove use of `small_rng` feature
- Minimal impact (code we rely on wasn't removed, just the feature gate)
- Reference: https://github.com/rust-random/rand/pull/1732
- Replace imports of `rand::Rng` with `rand::RngExt`
- Reference: https://github.com/rust-random/rand/pull/1717
- `StdRng` is no longer `Clone`, fix that somehow:
- Reference: https://github.com/rust-random/rand/pull/1677

https://github.com/apache/datafusion/blob/89e14f1de500e3f117574d91443b2c22520105c6/datafusion/common/benches/with_hashes.rs#L238

- `Fill` no longer implemented for slices
- Reference: https://github.com/rust-random/rand/pull/1652

https://github.com/apache/datafusion/blob/89e14f1de500e3f117574d91443b2c22520105c6/datafusion/functions/src/math/random.rs#L85

- `arrow-rs` still depends on `rand = "0.9"` and we interface with this, so we need to upgrade `arrow-rs`
- `create_primitive_array` comes from `arrow-rs`

https://github.com/apache/datafusion/blob/89e14f1de500e3f117574d91443b2c22520105c6/datafusion/functions-aggregate/benches/array_agg.rs#L91

### Describe alternatives you've considered

_No response_

### Additional context

See changelog for rand:

- https://github.com/rust-random/rand/blob/b1ad27346a9637979c2e286988b0aa55f771ecc9/CHANGELOG.md#0100---2026-02-08

Contributor guide

Open the contributing guide

Research direction

Start with datafusion/core/Cargo.toml and the workspace Cargo.toml entries for rand and rand_distr, then inspect the referenced uses in datafusion/common/benches/with_hashes.rs, datafusion/functions/src/math/random.rs, and datafusion/functions-aggregate/benches/array_agg.rs. Update the dependencies and resolve the listed rand API changes, including the arrow-rs compatibility issue. Done means the project builds with rand 0.10 and rand_distr 0.6 without the small_rng feature.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, build-system
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.