lnccbrown / lnccbrown/ssm-simulators

Custom `boundary` is accepted and silently ignored by 13 models whose compiled engine has no `boundary_fun` argument

Open
#310 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Jupyter Notebook
Stars
24
Forks
18
Avg merge
1d 14h
Merged PRs (30d)
9

Description

`Simulator(model, boundary=fn, boundary_params=[...])` builds a boundary dict and splats it into the compiled engine unconditionally. Eight `cssm` engines have no `boundary_fun` parameter, so the boundary is swallowed by `**kwargs` and discarded; the run completes normally using the engine's built-in boundary. 11 of the 13 affected models emit no warning at all, and for `ddm_legacy`/`addm` the returned `metadata["boundary_fun_type"]` reports the built-in boundary name. 97 of 113 built-in models honour a custom boundary correctly, and default usage (no boundary override) is unaffected.

Environment: ssm-simulators 0.13.2, Python 3.12, macOS (Darwin 25.4.0), `n_threads=1`.

## Reproducer

```python
"""ssms 0.13.2 -- custom boundary silently ignored by several compiled engines."""

import inspect

import numpy as np

from ssms.basic_simulators.simulator_class import Simulator
from ssms.config import get_model_registry

# Collapses to the 0.01 floor within ~100 ms. Any engine that honours it
# must return near-instant RTs.
def hard_collapse(t, a=1.0):
return np.maximum(a * np.exp(-50.0 * np.asarray(t)), 0.01)

def mean_rt(model, theta, **kw):
out = Simulator(model, **kw).simulate(theta=theta, n_samples=2000, max_t=10.0)
rts = np.asarray(out["rts"]).ravel()
return float(rts[rts != -999.0].mean())

BND = dict(boundary=hard_collapse, boundary_params=["a"])
ddm_th = {"v": 0.5, "a": 1.5, "z": 0.5, "t": 0.0}
lba_th = {"v0": 0.5, "v1": 0.3, "v2": 0.2, "a": 1.5, "z": 0.2, "theta": 0.0}

print("ddm default %.3f -> hard_collapse %.3f (honoured)"
% (mean_rt("ddm", ddm_th), mean_rt("ddm", ddm_th, **BND)))
print("ddm_legacy default %.3f -> hard_collapse %.3f (IGNORED)"
% (mean_rt("ddm_legacy", ddm_th), mean_rt("ddm_legacy", ddm_th, **BND)))
print("lba_angle_3 default %.3f -> hard_collapse %.3f (IGNORED)"
% (mean_rt("lba_angle_3", lba_th), mean_rt("lba_angle_3", lba_th, **BND)))
print("lba_angle_3 control: same model, a=1.5 -> a=0.3 gives %.3f "
"(engine does react to 'a', just not to boundary_fun)"
% mean_rt("lba_angle_3", {**lba_th, "a": 0.3}))

# ---- sweep every built-in model -------------------------------------------
ignored = []
for m in sorted(get_model_registry().list_models()):
cfg = get_model_registry().get(m)
th = dict(zip(cfg["params"], map(float, cfg["default_params"])))
base, coll = mean_rt(m, th), mean_rt(m, th, **BND)
if not (base - coll) / base > 0.10:
ignored.append(m)
print("\nboundary silently ignored by %d/%d built-in models:" % (
len(ignored), len(get_model_registry().list_models())))
print(ignored)

print("\nengine signatures:")
for f in ["ddm_flexbound", "ddm", "addm", "lba_vanilla", "lba_angle",
"rlwm_lba_pw_v1", "rlwm_lba_race", "poisson_race",
"racing_diffusion_model"]:
import cssm
print(" cssm.%-24s boundary_fun in signature: %s"
% (f, "boundary_fun" in inspect.signature(getattr(cssm, f)).parameters))

cfg = get_model_registry().get("lba_angle_3")
print("\nlba_angle_3 config: boundary_name=%r boundary=%r boundary_params=%r"
% (cfg["boundary_name"], cfg["boundary"].__name__, cfg.get("boundary_params")))
```

## Output

```
ddm default 1.889 -> hard_collapse 0.046 (honoured)
ddm_legacy default 0.556 -> hard_collapse 0.562 (IGNORED)
lba_angle_3 default 2.863 -> hard_collapse 2.872 (IGNORED)
lba_angle_3 control: same model, a=1.5 -> a=0.3 gives 0.379 (engine does react to 'a', just not to boundary_fun)
/…/site-packages/ssms/basic_simulators/simulator_class.py:651: UserWarning: Callable boundary expects parameters ['a'] but none were found in theta (keys: ['beta', 'q0', 'q1', 'deadline', 's']). The boundary function may receive missing arguments.
boundary_dict = make_boundary_dict(model_config_local, theta)
/…/site-packages/ssms/basic_simulators/simulator_class.py:651: UserWarning: Callable boundary expects parameters ['a'] but none were found in theta (keys: ['beta', 'q0', 'q1', 'q2', 'deadline', 's']). The boundary function may receive missing arguments.
boundary_dict = make_boundary_dict(model_config_local, theta)
/…/site-packages/ssms/basic_simulators/simulator_class.py:651: UserWarning: Callable boundary expects parameters ['a'] but none were found in theta (keys: ['beta', 'q0', 'q1', 'q2', 'q3', 'deadline', 's']). The boundary function may receive missing arguments.
boundary_dict = make_boundary_dict(model_config_local, theta)
/…/site-packages/ssms/basic_simulators/simulator_class.py:651: UserWarning: Callable boundary expects parameters ['a'] but none were found in theta (keys: ['r1', 'r2', 'k1', 'k2', 't', 'deadline', 's', 'r', 'k']). The boundary function may receive missing arguments.
boundary_dict = make_boundary_dict(model_config_local, theta)
/…/site-packages/ssms/basic_simulators/simulator_class.py:651: UserWarning: Callable boundary expects parameters ['a'] but none were found in theta (keys: ['v0', 'v1', 'v2', 'A', 'b', 't', 'deadline', 's', 'v']). The boundary function may receive missing arguments.
boundary_dict = make_boundary_dict(model_config_local, theta)

boundary silently ignored by 16/113 built-in models:
['addm', 'ddm_legacy', 'dev_rlwm_lba_pw_v1', 'dev_rlwm_lba_race_v1', 'dev_rlwm_lba_race_v2', 'inv_temp_softmax_2', 'inv_temp_softmax_3', 'inv_temp_softmax_4', 'lba2', 'lba3', 'lba4', 'lba_3_vs_constraint', 'lba_angle_3', 'lba_angle_3_vs_constraint', 'poisson_race', 'racing_diffusion_3']

engine signatures:
cssm.ddm_flexbound boundary_fun in signature: True
cssm.ddm boundary_fun in signature: False
cssm.addm boundary_fun in signature: False
cssm.lba_vanilla boundary_fun in signature: False
cssm.lba_angle boundary_fun in signature: False
cssm.rlwm_lba_pw_v1 boundary_fun in signature: False
cssm.rlwm_lba_race boundary_fun in signature: False
cssm.poisson_race boundary_fun in signature: True
cssm.racing_diffusion_model boundary_fun in signature: False

lba_angle_3 config: boundary_name='constant' boundary='constant' boundary_params=None
```

(Only the venv path prefix is elided above. `inv_temp_softmax_2/3/4` are choice-only simulators that return `PLACEHOLDER_RT = -1.0` for every trial — `ssms/basic_simulators/inv_temp_softmax.py:10` — so a flat mean RT there is trivially true; that takes the meaningful count from 16 to 13.)

Warning behaviour, checked separately with `warnings.simplefilter("always")` and `n_samples=200`:

```
addm warnings=NONE metadata boundary_fun_type='addm_collapse'
ddm_legacy warnings=NONE metadata boundary_fun_type='constant'
lba2 / lba3 / lba4 warnings=NONE metadata boundary_fun_type=''
lba_3_vs_constraint warnings=NONE metadata boundary_fun_type=''
lba_angle_3 warnings=NONE metadata boundary_fun_type=''
lba_angle_3_vs_constraint warnings=NONE metadata boundary_fun_type=''
dev_rlwm_lba_pw_v1 warnings=NONE metadata boundary_fun_type=''
dev_rlwm_lba_race_v1/v2 warnings=NONE metadata boundary_fun_type=''
poisson_race warnings=["Callable boundary expects parameters ['a'] but none were fou…"]
racing_diffusion_3 warnings=["Callable boundary expects parameters ['a'] but none were fou…"]
```

## Why

Python layer builds and forwards the boundary without checking that the engine accepts one:

- `ssms/basic_simulators/simulator_class.py:388-400` — `_apply_custom_boundary` sets `boundary`/`boundary_name`/`boundary_params` with no engine-capability check.
- `ssms/basic_simulators/simulator_class.py:651-667` — `simulate()` calls `make_boundary_dict` and splats the result into `model_config_local["simulator"](**theta, **boundary_dict, ...)`.
- `ssms/basic_simulators/simulator.py:285-339, 782-788` — same construction and unconditional splat on the functional path.
- `ssms/basic_simulators/simulator_class.py:321-358` — `_infer_simulator_requirements` already computes `supports_boundary = "boundary_fun" in params`, but it is never called anywhere in the package (`grep -rn _infer_simulator_requirements ssms/` returns only the `def` line).

Engines with no `boundary_fun` parameter, so it lands in `**kwargs` and is never read:

- `cssm/ddm_models.pyx:349-363` — `ddm` (model `ddm_legacy`); line 561 hardcodes `boundary_fun_name='constant'` into metadata.
- `cssm/addm_models.pyx:320-338` — `addm`; line 459 hardcodes `boundary_fun_name='addm_collapse'`.
- `cssm/lba_models.pyx:36-46, 144-156, 253-266, 384-396` — `lba_vanilla`, `lba_angle`, `rlwm_lba_pw_v1`, `rlwm_lba_race`. `grep -c boundary_fun cssm/lba_models.pyx` → `0`.
- `cssm/race_models.pyx:917-931` — `racing_diffusion_model`.
- `cssm/poisson_race_models.pyx:59` — declares `boundary_fun = None, # unused, kept for interface compatibility`, so signature introspection gives a false positive here.

Contrast: `cssm/ddm_models.pyx:589-606` (`ddm_flexbound`) declares `boundary_fun`/`boundary_params` and calls `compute_boundary` (`cssm/_utils.pyx:655-666`).

## Suggested fix

Refuse loudly in the Python layer rather than plumbing a boundary the engine cannot use. Wire up the existing dead code in `_apply_custom_boundary` (and the equivalent for `drift`):

```python
reqs = self._infer_simulator_requirements(config["simulator"])
if not reqs["supports_boundary"]:
raise ValueError(
f"Model {config['name']!r} uses simulator "
f"{config['simulator'].__name__!r}, which computes its boundary "
"internally and cannot accept a custom boundary function."
)
```

Signature introspection alone is not enough because of `poisson_race`, so prefer an explicit `"supports_custom_boundary": bool` on each model config (or a module-level set of engines that genuinely honour it), falling back to introspection when absent. The engines to mark `False` are `cssm.ddm`, `cssm.addm`, `cssm.lba_vanilla`, `cssm.lba_angle`, `cssm.rlwm_lba_pw_v1`, `cssm.rlwm_lba_race`, `cssm.racing_diffusion_model`, `cssm.poisson_race` (plus `cssm.full_ddm_hddm_base`, `ddm_models.pyx:55`, which no registry model currently routes to). Put the guard in or immediately after `make_boundary_dict` so `simulator.py:771` is covered too. The sweep above is directly reusable as a parametrised regression test over `get_model_registry().list_models()`: each model must either shorten mean RT measurably under a collapsing bound or raise.

## Is this intended?

Partly, and the issue concedes it: these engines deliberately compute their own boundary (`lba_angle` derives a linear collapse from `theta`; `addm` computes `±(a - b·t)` internally; `poisson_race:59` says so in a comment). The complaint is not that they lack a general boundary hook — it is that the Python layer accepts, validates and forwards a boundary to them with no check, when the check already exists as dead code.

Two things the report explicitly does not claim: the value of `a` is not dropped (`make_boundary_dict` does route it into `boundary_params`, and `a` is also passed independently — the `a=1.5 → 0.3` control moves mean RT 2.863 → 0.379); and "no warning at all" is not universally true, since 5 of the 16 do warn — but incidentally, with the text "expects parameters ['a'] but none were found in theta", which is a missing-parameter complaint, not "this boundary will be ignored".

Unrelated observation while measuring: the LBA engines appear to ignore `random_state` (repeat `simulate(..., random_state=42)` calls give mean RTs differing by ~0.3%). That wobble is why the ignored rows above are not bit-identical; the honouring models move 60-98% over the same comparison, well clear of it.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in ssms/basic_simulators/simulator_class.py at _apply_custom_boundary, _infer_simulator_requirements, and the simulate() boundary splat, then trace the equivalent path in ssms/basic_simulators/simulator.py. Run the supplied model-registry sweep as a regression check. Done means unsupported custom boundaries are rejected on both paths while supported engines continue to honor them.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.