AllenNeuralDynamics / AllenNeuralDynamics/aind-dynamic-foraging-bfm-dispatcher

[Cross-species transfer] Run study 09 benchmark and report transfer curves

Open
#126 5 comments 0 reactions 1 assignee Claimed by @hanhou View on GitHub
baselines evaluation extension priority:P2
Dominant language
Python
Stars
0
Forks
0
Avg merge
2h 9m
Merged PRs (30d)
31

Description

## Context

Parent: #32. The original Grossman, Chen, and Zid matched-half benchmark, author-aligned baselines, representative sessions, and embedding-space analysis were merged in #130. The ten-cohort Stage-A expansion is complete in draft PR #139.

## Stage-A expansion

- Draft implementation and provenance PR: #139
- Admitted with existing schemas: 10 new cohorts (9 v1, 1 v2), for 13 total with the three frozen starter cohorts.
- Skipped without adding schema v3: López-Yépez human, Shin, Alsiö cohort VI, Hattori, and Samejima; exact reasons are in the committed dataset survey.
- [x] Pin full-release sources, licenses/terms, checksums, inclusion rules, and exact cohort counts.
- [x] Generate deterministic v1/v2 manifests and pass exact-count, binary-choice/reward, and loader-contract validation.
- [x] Pass one real-subject local CPU Q smoke for v1 and v2.
- [x] Pass one D=614, seed-0 embedding-only GPU smoke for every newly admitted cohort.
- [x] Complete and freeze the 150-cell expansion GRU matrix on Beaker GPU (195/195 cells across all 13 cohorts).
- [ ] Complete all ten expansion common-Q fits on Allen HPC CPU as SLURM array `25581304`.
- [ ] Freeze exact ordered GRU/Q held-out trial-key parity and update Results 2 and 3.
- [ ] Review Stage-A results before selecting any new author model.

Draft PR #139 contains the completed common-Q fits, frozen parity evidence, and reports; those acceptance boxes remain open until the PR is merged.

## Stage-A decision result

At D=614, the unadjusted subject-paired result favors GRU for Grossman, Chen, Lebedeva, and López-Yépez mouse; it favors common Q for Zid, Kwak, Miller, Findling, and Eckstein; Beron, Tang, Alsiö, and Costa are unresolved. Every cohort improves in trial-pooled GRU likelihood from D=10 to D=614, although several curves peak earlier. Every external cohort is farther from the source embedding distribution than held-out AIND mice in all three seeds.

## Reports

- Result 2: `studies/09-gru-cross-species-transfer/analysis/reports/r2-author-aligned-baselines.md`
- Result 3: `studies/09-gru-cross-species-transfer/analysis/reports/r3-embedding-space.md`

## Remaining original scope

- [ ] Grossman/Chen K curves.
- [ ] Chance/bias and source-informed HB-Q controls when available.
- [ ] Full metric/bootstrap package.

Stage A does not add schema v3 or implement a new author-selected model.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.