hans / hans/barakeet

Sign-flip permutation-null significance band for the contrast_plot acoustic panel

Open
#6 0 comments 0 reactions 0 assignees View on GitHub
ready-for-agent
Dominant language
Jupyter Notebook
Stars
0
Forks
0
PR merge metrics
No merged PRs in 30d

Description

## Problem Statement

The acoustic panel of the continuous-time contrast plot
(`notebooks/causal46_joined/contrast_plot.py`, rules `contrast_plot` and
`contrast_plot_per_pair`) draws, per `conjunction_category`, the grand mean over
cells of the ambiguous-trial acoustic-step contrast (`b4_by_acoustic`, s_hi − s_lo
among qualifying ambiguous steps, behavior-balanced 50/50 per step,
within-completion), after orienting each cell by `endpoint_sign` — sign(median(step
6 − step 1)) from the site's *unambiguous endpoint* window, gated on
`best_ci_aligned_excludes_zero`. A positive grand mean means the ambiguous acoustic
contrast runs in the same direction as the clean endpoint tuning — an out-of-sample
test of acoustic-code consistency.

Unlike the behavioral panel, the orientation here is computed on endpoint trials
that are **disjoint** from the ambiguous trials being plotted, so there is no
self-orientation / rectification artifact and the null expectation is genuinely
zero. But the panel currently shows only a cross-cell SEM ribbon against an implicit
zero baseline and no significance test, so the consistency claim ("aligned beyond
chance") is asserted rather than demonstrated.

## Solution

Add a **per-cell sign-flip null** to the acoustic panel. Holding the observed
ambiguous contrasts fixed, randomly flip each cell's `endpoint_sign` ∈ {+1, −1},
re-average across the group, and repeat R times to form a null band. Under the null
"alignment between endpoint tuning and the ambiguous contrast is chance," each
cell's orientation is equally likely concordant or discordant, so the band is
centered at **zero**. Shade the band per category and mark where the observed line
exits it; the observed excess over the zero-centered band is the consistency test.

The oriented grand mean `mean_c[ endpoint_sign_c · contrast_c(t) ]` is already an
alignment statistic (positive only when the ambiguous contrast is, on average,
sign-concordant with endpoint tuning), so the sign-flip permutation is exactly its
matched null — the nonparametric one-sample-against-zero test on the per-cell
aligned trajectories. It operates on the per-cell trajectories directly; no epoch
reload.

## User Stories

1. As a researcher, I want the acoustic-panel curves drawn with a shaded null band,
so that I can see whether the endpoint-aligned ambiguous contrast is beyond
chance rather than eyeballing it against zero.
2. As a researcher, I want the null produced by flipping each cell's endpoint
orientation sign (not by shuffling anything across cells or reloading trials),
so that it holds the observed ambiguous contrasts fixed and tests only the
sign relationship.
3. As a researcher, I want the null band centered at zero, so that the panel
correctly reflects that its orientation is out-of-sample and carries no
rectification floor (in explicit contrast to the behavioral panel).
4. As a researcher, I want a timepoint marked significant only where the observed
curve exits the sign-flip band, so that acoustic-consistency significance is
shown at the right temporal resolution.
5. As a researcher, I want cells whose ambiguous contrast is anti-aligned or
randomly aligned with endpoint tuning to pull the observed curve toward the band,
so that the statistic genuinely measures consistency and not mere effect
presence.
6. As a researcher, I want the existing selection preserved exactly — the endpoint
reliability gate (`best_ci_aligned_excludes_zero`) and the `conjunction_category`
grouping — so that the panel keeps showing the same cells.
7. As a researcher, I want the band reproducible under a fixed seed and the
permutation count to be a Snakefile-wired notebook parameter, so that I can trade
speed for resolution.
8. As a researcher, I want the band added to both `contrast_plot` and
`contrast_plot_per_pair`, so that all published variants carry the same test.
9. As a researcher, I want the panel's methods note to state that — unlike the
behavioral panel — the acoustic panel's selection (endpoint reliability +
morphology) is on different data than the tested quantity (ambiguous-trial
alignment), so that its significance is read as non-circular.
10. As a maintainer, I want the sign-flip null to reuse the same
`oriented_group_band` seam as the behavioral panel, differing only in a
`null_mode`, so that both panels share one tested code path.
11. As a maintainer, I want a test proving the sign-flip band is centered at zero on
random-orientation data, so that the "no rectification floor here" property is
guarded.
12. As a maintainer, I want a test proving consistently-aligned synthetic cells push
the observed curve outside the band while randomly-aligned cells keep it inside,
so that the consistency calibration is verified behaviorally.

## Implementation Decisions

- **Same seam as the behavioral panel.** Extend the `oriented_group_band(...)`
helper in `_contrast.py` (introduced by the behavioral-panel work) with a
`null_mode` selector:
- `behavior_permute` (behavioral panel): permute behavior labels within cell +
recompute sign under permutation → band at the rectification floor.
- `sign_flip` (this panel): hold observed contrasts fixed, flip each cell's
orientation sign per replicate → band centered at zero.
If the behavioral-panel helper does not exist yet, this work introduces the shared
helper with both modes.
- **Null unit = the per-cell oriented trajectory.** The sign-flip operates on the
already-computed per-cell ambiguous contrast (`b4_by_acoustic` bootstrap mean per
cell) times its `endpoint_sign`; no trial-level recompute, no epoch reload.
- **Orientation source unchanged.** `endpoint_sign` from
`per_cell_best(a_per_window_all, …)` with the `best_ci_aligned_excludes_zero`
gate, exactly as today. The change is additive (a band), not a reorientation.
- **Selection unchanged.** Endpoint reliability gate and `conjunction_category`
grouping preserved.
- **Parameters.** Reuse `n_perm` and `null_seed` (shared with the behavioral panel)
in the notebook parameters cell and the `run_notebook(parameters=dict(...))` calls
for both rules. No type annotations in the parameters cell; every parameters-cell
name must appear in the Snakefile call.
- **Scope.** Both `contrast_plot` and `contrast_plot_per_pair`. The behavioral panel
is out of scope here (its own ticket).
- **Significance display.** Two-sided band from percentiles of the sign-flip null
matrix; mark where the observed curve exits it. Rendering detail left to
implementation but consistent with the behavioral panel's band styling.

## Testing Decisions

- **Good test = external behavior at the seam.** Target the `sign_flip` mode of
`oriented_group_band` with synthetic per-cell trajectories + signs; not the
notebook, not the parquet artifacts.
- **Modules tested:** `_contrast.py`.
- **Cases:**
1. Random-orientation cells (sign independent of contrast) → observed grand mean
lies inside the band, and the band is centered at **zero** (no floor).
2. Consistently-aligned cells (contrast sign-concordant with `endpoint_sign` in a
known window) → observed curve exits the band in that window.
3. Anti-aligned cells → observed curve exits the band on the *negative* side
(the statistic is signed, not rectified).
4. Determinism under fixed `seed`.
- **Prior art:** the sign-flip / permutation tests elsewhere in the causal46_joined
suite; share fixture style with the behavioral-panel band tests.

## Out of Scope

- The **behavioral** panel and its behavior-permutation (floor-centered) null —
separate ticket.
- The **step-permutation** alternative null (permute s_hi/s_lo within behavior ×
word_end to test acoustic-effect *presence + alignment*). Deliberately not chosen:
the panel's claim is pure alignment/consistency, and sign-flip tests exactly that
without an epoch reload.
- Changing the orientation source, the endpoint reliability gate, or the
`conjunction_category` definitions.
- Multiple-comparison correction across timepoints; this is a visualization aid.

## Further Notes

- **Non-circularity (state in the panel docstring).** The acoustic panel's selection
(endpoint-window reliability + early morphology type) is on different data than the
tested quantity (ambiguous-trial alignment), so the sign-flip stars are genuinely
non-circular — no "self-fulfilling" caveat like the behavioral panel. One check: if
the early-type annotation (`type1_acoustic_only` etc.) is itself derived from
ambiguous data, there is residual conditioning; if it is an endpoint/acoustic call,
the panel is clean.
- The acoustic panel's *existing* SEM-against-zero ribbon is already roughly valid
(unlike the behavioral panel's, whose floor ≠ 0); the sign-flip null makes the test
exact/robust and puts both panels on the same footing.
- Sibling of the behavioral-panel spec; both land on one `oriented_group_band` seam
with a `null_mode` switch. Background audit: finding M2 in
`docs/superpowers/plans/2026-07-10-causal46-joined-scientific-audit.md`.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in _contrast.py at the oriented_group_band helper and review the existing behavioral-panel band tests. Trace how contrast_plot and contrast_plot_per_pair pass notebook parameters through the Snakefile, then verify both acoustic panels use the sign-flip band with fixed-seed behavior and that the seam tests cover random, aligned, anti-aligned, and deterministic cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook, python
Domain
data-visualization, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
64/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.