lnccbrown / lnccbrown/ssm-simulators

UniformParameterSampler draws parameters in a hash-dependent order, so a seeded sample is not reproducible across processes

Open Beginner friendly
#358 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Jupyter Notebook
Stars
24
Forks
18
Avg merge
1d 14h
Merged PRs (30d)
9

Description

## Describe the bug

`UniformParameterSampler` (via `AbstractParameterSampler._build_dependency_graph` / `_topological_sort`) iterates a `set` when it orders the parameters it draws, so the order in which the RNG is consumed depends on `PYTHONHASHSEED`. With the same `rng` state, two processes can draw different parameter vectors: the samples are only reproducible within one interpreter session.

Found while making LANfactory's derived corpora byte-reproducible (`lanfactory.derive`, unmerged): a corpus file generated from `np.random.default_rng([seed, i])` differed between `PYTHONHASHSEED=0` and `PYTHONHASHSEED=1` runs. LANfactory now draws in `model_config["params"]` order on its side, so it no longer depends on this, but every other consumer of the sampler (the simulation pipeline included) inherits the non-determinism.

## To Reproduce

```bash
for h in 0 1; do PYTHONHASHSEED=$h python - <<'PY'
import numpy as np
from ssms.config import ModelConfigBuilder
from ssms.dataset_generators.parameter_samplers import UniformParameterSampler
cfg = ModelConfigBuilder.from_model("ddm")
s = UniformParameterSampler(param_space=cfg["param_bounds_dict"], constraints=ModelConfigBuilder.get_sampling_transforms(cfg))
out = s.sample(n_samples=3, rng=np.random.default_rng(0))
print({k: np.round(v, 4).tolist() for k, v in out.items()})
PY
done
```

## Expected behavior

The same seed gives the same sample in every process: order the parameters deterministically (e.g. `sorted(...)`, or the config's `params` order) before drawing.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in ssms/dataset_generators/parameter_samplers.py, focusing on AbstractParameterSampler._build_dependency_graph and _topological_sort, then run the provided PYTHONHASHSEED reproduction. Ensure parameter ordering does not depend on set iteration, and verify that identical RNG seeds produce the same sample output across processes.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
data
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
85/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.