lnccbrown / lnccbrown/HSSM

ddm: approx_differentiable z bound (0.0, 1.0) is wider than the network's training box [0.1, 0.9]

Open
#1,229 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
124
Forks
24
Avg merge
19h 32m
Merged PRs (30d)
60

Description

## Summary

For `ddm`, the `approx_differentiable` likelihood declares `z: (0.0, 1.0)`, but `ddm.onnx` was trained on `z ∈ [0.1, 0.9]` (ssms `model_config["ddm"]["param_bounds"]`). **20% of the declared interval is outside the training box.** The LAN does not fail there — it extrapolates and returns a finite, plausible, wrong log-density, which the sampler will explore freely.

This affects the most-used model in the ecosystem.

## Evidence

Same `v=0.5, a=1.2, t=0.2`, four RTs, varying only `z`. Max absolute difference between `ddm.onnx` and `logp_ddm` (the exact analytical density), in log-likelihood units per trial:

| z | in training box | max |LAN − analytical| |
|---|---|---|
| 0.05 | **no** | 0.397 |
| 0.10 | yes | 0.107 |
| 0.20 | yes | 0.021 |
| 0.50 | yes | 0.092 |
| 0.80 | yes | 0.043 |
| 0.90 | yes | 0.095 |
| 0.95 | **no** | 0.379 |
| 0.99 | **no** | **1.677** |

Inside the trained range the approximation holds to ~0.1 log units. Outside it, error grows 4–16×. At `z = 0.99` that is a factor of ~5 in likelihood **per trial**; across a few hundred trials the posterior is being shaped by a density the network never learned.

Repro:

```python
import numpy as np, onnxruntime as ort
from huggingface_hub import hf_hub_download
from hssm.likelihoods.analytical import logp_ddm
import pytensor, pytensor.tensor as pt

sess = ort.InferenceSession(hf_hub_download("franklab/HSSM", "ddm.onnx"))
iname = sess.get_inputs()[0].name
rt = np.array([0.4, 0.7, 1.0, 1.5], dtype=np.float32)
resp = np.array([1.0, -1.0, 1.0, -1.0], dtype=np.float32)
v, a, t = 0.5, 1.2, 0.2

d = pt.matrix(); vv, aa, zz, tt = pt.scalars("v", "a", "z", "t")
f = pytensor.function([d, vv, aa, zz, tt], logp_ddm(d, vv, aa, zz, tt))
for z in (0.05, 0.5, 0.99):
lan = np.array([sess.run(None, {iname: np.array([[v, a, z, t, r, c]], dtype=np.float32)})[0].item()
for r, c in zip(rt, resp)])
ana = np.asarray(f(np.column_stack([rt, resp]).astype("float64"), v, a, z, t)).reshape(-1)
print(z, np.max(np.abs(lan - ana)))
```

## Why it is probably a copy-over

`z: (0.0, 1.0)` is declared identically for all three of `ddm`'s likelihood kinds. For `analytical` and `blackbox` it is correct — `z` is a proportion and the closed-form density is valid across the interval. In the `approx_differentiable` block, `v`, `a` and `t` all match the training box exactly and only `z` does not, which reads like the analytical bound was carried over rather than derived from the network.

## Why it matters in practice

A start-point bias near the edges is not exotic — response bias and speed/accuracy manipulations produce it routinely. A genuinely biased participant sits right at `z ≈ 0.1` or `0.9`, precisely where the trained region ends, and the sampler will push past it during tuning. There is no warning: `initial_logp` is finite, no gate fires, and the fit completes normally.

## Options

1. **Narrow the declared bound to `(0.1, 0.9)`.** One line, matches the other seven models, and is what the bound is supposed to mean. It changes user-facing sampling behaviour and could alter previously published fits, which is why this is an issue rather than a PR.
2. **Retrain `ddm.onnx` on a wider `z` box.** Removes the constraint rather than enforcing it, but costs a training run and a new artifact.
3. **Warn instead of bound.** Keep `(0.0, 1.0)` and emit a warning when the posterior mass strays outside the trained region. Preserves current behaviour, surfaces the risk.

I lean toward (1): the bound documents what the network knows, and silently permitting extrapolation is the worse failure mode. But it is a behaviour change on the most-used model and belongs to the maintainers.

## Context

Found while auditing all eight models that carry `approx_differentiable` bounds against their ssms training boxes. Six matched exactly. Two did not:

- `ddm_sdv` / `sv`: declared `(0.0, 1.0)`, trained `(0.001, 2.5)` — *narrower* than trained, so safe but withholding validated range. Fixed separately, along with a regression test.
- `ddm` / `z`: this issue — *wider* than trained.

That test (`tests/unit/modelconfig/test_lan_bounds_match_training.py`) asserts the mismatch set **equals** `{("ddm", "z")}` rather than merely containing it, so a new drift fails and so does resolving this one without removing the waiver. Closing this issue means deleting that entry.

Contributor guide

Open the contributing guide

Research direction

Start with the ddm approx_differentiable model configuration and compare its z bound with ssms model_config["ddm"]["param_bounds"]. Run tests/unit/modelconfig/test_lan_bounds_match_training.py and review its expected mismatch set; done means the chosen resolution is reflected in the configuration and the regression test no longer requires this waiver.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.