lnccbrown / lnccbrown/HSSM

Failing parameter recovery on truncated data

Open
#816 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
124
Forks
24
Avg merge
19h 32m
Merged PRs (30d)
60

Description

I have a dataset for an experiment where the response window timed out at a fixed time, which means that on approximately 15% of trials, response and rt are unrecorded. I want to use a DDM to model participant responses, but am a little confused as to whether the `missing` or `deadline` conditions of HSSM is correct to use for this.

As a test, I simulated a dataset using HSSM, and modelled it four ways. First, I used the whole dataset. Second, I truncated it by deleting trials for which the rt was above the 85th percentile. Third, a version where those long trials were marked as -999 and the argument `missing=True` was passed to HSSM, and a version with an added column, `deadline` filled with the value of the 85th percentile and `deadline=True` passed to HSSM. I used the CPN and OPN supplied by @digicosmos in this thread [https://github.com/lnccbrown/HSSM/issues/173](url).

The parameter estimates for each of these models are summarized in this Forest plot. You can see that the estimates from `missing_data` and `deadline` are essentially no different to those when I just delete the long responses. Code to reproduce this is below. I'd appreciate any advice as to whether I'm using the package wrong or whether this is a bug.

Image

```python
import hssm
import matplotlib.pyplot as plt
import arviz as az

# Simulate data
v_true, a_true, z_true, t_true = [0.5, 1.5, 0.5, 0.5]
dataset = hssm.simulate_data(
model="ddm",
theta=[v_true, a_true, z_true, t_true],
size=1000,
)

truncated_del = dataset[dataset["rt"] < 3.9] # Approximate 85% quantile

truncated_missing = dataset.copy()
truncated_missing.loc[truncated_missing["rt"] >= 3.9, "rt"] = -999
truncated_missing["deadline"] = 3.9

truncated_deadline = dataset.copy()
truncated_deadline.loc[truncated_missing["rt"] == -999, "rt"] = 3.9

# Create and sample models
full_model = hssm.HSSM(
model="ddm",
data=dataset,
)

truncated_model = hssm.HSSM(
model="ddm",
data=truncated_del,
)

missing_model = hssm.HSSM(
model="ddm",
data=truncated_missing,
missing_data=-999,
loglik_missing_data=r"C:\Users\ddgpa\Downloads\ddm_cpn (1).onnx",
)
deadline_model = hssm.HSSM(
model="ddm",
data=truncated_missing,
deadline="deadline",
loglik_missing_data=r"C:\Users\ddgpa\Downloads\ddm_opn.onnx",
)

full_idata = full_model.sample()
truncated_idata = truncated_model.sample()
missing_idata = missing_model.sample()
deadline_idata = deadline_model.sample()

# Plot
fig, ax = plt.subplots(figsize=(9, 5))
az.plot_forest(
[full_idata, truncated_idata, missing_idata, deadline_idata],
model_names=["Full", "Simple", "Missing", "Deadline"],
combined=True,
ax=ax,
)
```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the four models built with hssm.simulate_data and HSSM(...), using the supplied CPN and OPN files and comparing their sampled parameter estimates. Trace how missing_data=-999 and deadline are handled during sample(); done means explaining whether the similar estimates are expected or identifying a reproducible implementation defect.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.