ddm: approx_differentiable z bound (0.0, 1.0) is wider than the network's training box [0.1, 0.9]
- Dominant language
- Python
- Stars
- 124
- Forks
- 24
- Avg merge
- 19h 32m
- Merged PRs (30d)
- 60
Description
## Summary
For `ddm`, the `approx_differentiable` likelihood declares `z: (0.0, 1.0)`, but `ddm.onnx` was trained on `z ∈ [0.1, 0.9]` (ssms `model_config["ddm"]["param_bounds"]`). **20% of the declared interval is outside the training box.** The LAN does not fail there — it extrapolates and returns a finite, plausible, wrong log-density, which the sampler will explore freely.
This affects the most-used model in the ecosystem.
## Evidence
Same `v=0.5, a=1.2, t=0.2`, four RTs, varying only `z`. Max absolute difference between `ddm.onnx` and `logp_ddm` (the exact analytical density), in log-likelihood units per trial:
| z | in training box | max |LAN − analytical| |
|---|---|---|
| 0.05 | **no** | 0.397 |
| 0.10 | yes | 0.107 |
| 0.20 | yes | 0.021 |
| 0.50 | yes | 0.092 |
| 0.80 | yes | 0.043 |
| 0.90 | yes | 0.095 |
| 0.95 | **no** | 0.379 |
| 0.99 | **no** | **1.677** |
Inside the trained range the approximation holds to ~0.1 log units. Outside it, error grows 4–16×. At `z = 0.99` that is a factor of ~5 in likelihood **per trial**; across a few hundred trials the posterior is being shaped by a density the network never learned.
Repro:
```python
import numpy as np, onnxruntime as ort
from huggingface_hub import hf_hub_download
from hssm.likelihoods.analytical import logp_ddm
import pytensor, pytensor.tensor as pt
sess = ort.InferenceSession(hf_hub_download("franklab/HSSM", "ddm.onnx"))
iname = sess.get_inputs()[0].name
rt = np.array([0.4, 0.7, 1.0, 1.5], dtype=np.float32)
resp = np.array([1.0, -1.0, 1.0, -1.0], dtype=np.float32)
v, a, t = 0.5, 1.2, 0.2
d = pt.matrix(); vv, aa, zz, tt = pt.scalars("v", "a", "z", "t")
f = pytensor.function([d, vv, aa, zz, tt], logp_ddm(d, vv, aa, zz, tt))
for z in (0.05, 0.5, 0.99):
lan = np.array([sess.run(None, {iname: np.array([[v, a, z, t, r, c]], dtype=np.float32)})[0].item()
for r, c in zip(rt, resp)])
ana = np.asarray(f(np.column_stack([rt, resp]).astype("float64"), v, a, z, t)).reshape(-1)
print(z, np.max(np.abs(lan - ana)))
```
## Why it is probably a copy-over
`z: (0.0, 1.0)` is declared identically for all three of `ddm`'s likelihood kinds. For `analytical` and `blackbox` it is correct — `z` is a proportion and the closed-form density is valid across the interval. In the `approx_differentiable` block, `v`, `a` and `t` all match the training box exactly and only `z` does not, which reads like the analytical bound was carried over rather than derived from the network.
## Why it matters in practice
A start-point bias near the edges is not exotic — response bias and speed/accuracy manipulations produce it routinely. A genuinely biased participant sits right at `z ≈ 0.1` or `0.9`, precisely where the trained region ends, and the sampler will push past it during tuning. There is no warning: `initial_logp` is finite, no gate fires, and the fit completes normally.
## Options
1. **Narrow the declared bound to `(0.1, 0.9)`.** One line, matches the other seven models, and is what the bound is supposed to mean. It changes user-facing sampling behaviour and could alter previously published fits, which is why this is an issue rather than a PR.
2. **Retrain `ddm.onnx` on a wider `z` box.** Removes the constraint rather than enforcing it, but costs a training run and a new artifact.
3. **Warn instead of bound.** Keep `(0.0, 1.0)` and emit a warning when the posterior mass strays outside the trained region. Preserves current behaviour, surfaces the risk.
I lean toward (1): the bound documents what the network knows, and silently permitting extrapolation is the worse failure mode. But it is a behaviour change on the most-used model and belongs to the maintainers.
## Context
Found while auditing all eight models that carry `approx_differentiable` bounds against their ssms training boxes. Six matched exactly. Two did not:
- `ddm_sdv` / `sv`: declared `(0.0, 1.0)`, trained `(0.001, 2.5)` — *narrower* than trained, so safe but withholding validated range. Fixed separately, along with a regression test.
- `ddm` / `z`: this issue — *wider* than trained.
That test (`tests/unit/modelconfig/test_lan_bounds_match_training.py`) asserts the mismatch set **equals** `{("ddm", "z")}` rather than merely containing it, so a new drift fails and so does resolving this one without removing the waiver. Closing this issue means deleting that entry.
Contributor guide
Research direction
Start with the ddm approx_differentiable model configuration and compare its z bound with ssms model_config["ddm"]["param_bounds"]. Run tests/unit/modelconfig/test_lan_bounds_match_training.py and review its expected mismatch set; done means the chosen resolution is reflected in the configuration and the regression test no longer requires this waiver.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 58/100