AllenNeuralDynamics / AllenNeuralDynamics/aind-dynamic-foraging-data-utils

docDB/Code Ocean: multiple processed assets per (subject, date) for some dynamic-foraging sessions

Abierto
#146 1 comentario 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Python
Estrellas
2
Forks
0
Merge medio
1 d 10 h
PR fusionados (30 d)
3

Descripción

## Summary

When enumerating **all processed dynamic-foraging sessions** from docDB (filter on
`session.*.software.name == "dynamic-foraging-task"` + `data_description.data_level == "derived"`),
**446 of 15822** `(subject_id, session_date)` day-groups have **more than one**
processed behavior asset (16338 sessions total). These are a **mix** of genuinely separate
same-day sessions and likely same-session artifacts, so a session table cannot safely be keyed by
`(subject_id, session_date)` alone — the full `session_name` (incl. `HH-MM-SS`) is needed as identity.

Han's pipeline session table, by contrast, has **at most one session per `(subject, date)`**, so this
is specific to the Code Ocean / docDB inventory.

## Sessions-per-(subject, date) distribution

| sessions in a day | # of (subject,date) |
|---|---|
| 1 | 15376 |
| 2 | 389 |
| 3 | 49 |
| 4 | 6 |
| 5 | 1 |
| 8 | 1 |

## Gap between same-day sessions (minutes)

min=0, 25%=6, median=23, 75%=50, max=738
- **< 5 min apart: 96 days** — likely the *same* session (restart / re-acquire / artifact)
- **>= 30 min apart: 184 days** — likely *genuinely separate* sessions (e.g. morning + afternoon)

## Examples

| subject | date | session times (HH-MM-SS) |
|---|---|---|
| `826162` | `2026-01-08` | 11-23-15, 12-46-16, 13-04-20, 13-16-57, 13-22-19, 13-24-59, 13-26-56, 13-28-41 |
| `712634` | `2024-05-29` | 11-10-52, 17-50-00, 19-36-02 |
| `731302` | `2024-10-08` | 13-59-25, 14-02-48 |

(e.g. the first is hours apart — clearly distinct sessions; the last is minutes apart — likely one session.)

## Why it matters

The foraging parquet-cache builder (`foraging_cache/`) unions Han's session table with the docDB/CO
universe. Matching is done on `(subject_id, session_date)` (Han's `nwb_suffix`/HHMMSS often differs
from docDB's for the same session), but sessions must **not** be collapsed by `(subject, date)` or the
~184+ genuinely-separate same-day sessions would be lost. Tracking here so we
decide how same-day duplicates (esp. the <5-min artifact pairs) should be handled / de-duplicated.

## Reproduce

```python
import pandas as pd, re
from aind_data_access_api.document_db import MetadataDbClient

client = MetadataDbClient(host="api.allenneuraldynamics.org",
database="metadata_index", collection="data_assets")
records = client.retrieve_docdb_records(
filter_query={
"$or": [
{"session.data_streams.software.name": "dynamic-foraging-task"},
{"session.stimulus_epochs.software.name": "dynamic-foraging-task"},
],
"data_description.data_level": "derived",
},
projection={"_id": 0, "name": 1},
paginate=True, paginate_batch_size=5000, # ~19k records; pagination required
)
df = pd.DataFrame(records)

# keep the behavior-processed NWB asset per session (drop PoseTracking / opto-sorted derived assets)
df["session_name"] = df["name"].str.extract(r"^(behavior_\d+_\d{4}-\d{2}-\d{2}_\d{2}-\d{2}-\d{2})_processed")
df = df.dropna(subset=["session_name"]).drop_duplicates("session_name")
df[["subj","date","tm"]] = df["session_name"].str.extract(r"behavior_(\d+)_(\d{4}-\d{2}-\d{2})_(\d{2}-\d{2}-\d{2})$")

per_day = df.groupby(["subj","date"]).size()
print(per_day.value_counts().sort_index()) # distribution
print(per_day[per_day > 1]) # the multi-session days
```

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.