AllenNeuralDynamics / AllenNeuralDynamics/aind-dynamic-foraging-data-utils

docDB/Code Ocean: multiple processed assets per (subject, date) for some dynamic-foraging sessions

Aberta
#146 1 comentário 0 reações 0 responsáveis Ver no GitHub
Linguagem predominante
Python
Estrelas
2
Forks
0
Merge médio
1d 10h
PRs com merge (30d)
3

Descrição

## Summary

When enumerating **all processed dynamic-foraging sessions** from docDB (filter on
`session.*.software.name == "dynamic-foraging-task"` + `data_description.data_level == "derived"`),
**446 of 15822** `(subject_id, session_date)` day-groups have **more than one**
processed behavior asset (16338 sessions total). These are a **mix** of genuinely separate
same-day sessions and likely same-session artifacts, so a session table cannot safely be keyed by
`(subject_id, session_date)` alone — the full `session_name` (incl. `HH-MM-SS`) is needed as identity.

Han's pipeline session table, by contrast, has **at most one session per `(subject, date)`**, so this
is specific to the Code Ocean / docDB inventory.

## Sessions-per-(subject, date) distribution

| sessions in a day | # of (subject,date) |
|---|---|
| 1 | 15376 |
| 2 | 389 |
| 3 | 49 |
| 4 | 6 |
| 5 | 1 |
| 8 | 1 |

## Gap between same-day sessions (minutes)

min=0, 25%=6, median=23, 75%=50, max=738
- **< 5 min apart: 96 days** — likely the *same* session (restart / re-acquire / artifact)
- **>= 30 min apart: 184 days** — likely *genuinely separate* sessions (e.g. morning + afternoon)

## Examples

| subject | date | session times (HH-MM-SS) |
|---|---|---|
| `826162` | `2026-01-08` | 11-23-15, 12-46-16, 13-04-20, 13-16-57, 13-22-19, 13-24-59, 13-26-56, 13-28-41 |
| `712634` | `2024-05-29` | 11-10-52, 17-50-00, 19-36-02 |
| `731302` | `2024-10-08` | 13-59-25, 14-02-48 |

(e.g. the first is hours apart — clearly distinct sessions; the last is minutes apart — likely one session.)

## Why it matters

The foraging parquet-cache builder (`foraging_cache/`) unions Han's session table with the docDB/CO
universe. Matching is done on `(subject_id, session_date)` (Han's `nwb_suffix`/HHMMSS often differs
from docDB's for the same session), but sessions must **not** be collapsed by `(subject, date)` or the
~184+ genuinely-separate same-day sessions would be lost. Tracking here so we
decide how same-day duplicates (esp. the <5-min artifact pairs) should be handled / de-duplicated.

## Reproduce

```python
import pandas as pd, re
from aind_data_access_api.document_db import MetadataDbClient

client = MetadataDbClient(host="api.allenneuraldynamics.org",
database="metadata_index", collection="data_assets")
records = client.retrieve_docdb_records(
filter_query={
"$or": [
{"session.data_streams.software.name": "dynamic-foraging-task"},
{"session.stimulus_epochs.software.name": "dynamic-foraging-task"},
],
"data_description.data_level": "derived",
},
projection={"_id": 0, "name": 1},
paginate=True, paginate_batch_size=5000, # ~19k records; pagination required
)
df = pd.DataFrame(records)

# keep the behavior-processed NWB asset per session (drop PoseTracking / opto-sorted derived assets)
df["session_name"] = df["name"].str.extract(r"^(behavior_\d+_\d{4}-\d{2}-\d{2}_\d{2}-\d{2}-\d{2})_processed")
df = df.dropna(subset=["session_name"]).drop_duplicates("session_name")
df[["subj","date","tm"]] = df["session_name"].str.extract(r"behavior_(\d+)_(\d{4}-\d{2}-\d{2})_(\d{2}-\d{2}-\d{2})$")

per_day = df.groupby(["subj","date"]).size()
print(per_day.value_counts().sort_index()) # distribution
print(per_day[per_day > 1]) # the multi-session days
```

Guia de contribuição

Nenhum guia de contribuição indexado para este repositório

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.