AllenNeuralDynamics / AllenNeuralDynamics/aind-dynamic-foraging-data-utils

docDB/Code Ocean: multiple processed assets per (subject, date) for some dynamic-foraging sessions

Aperta
#146 1 commento 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Python
Stelle
2
Fork
0
Merge medio
1g 10h
PR unite (30g)
3

Descrizione

## Summary

When enumerating **all processed dynamic-foraging sessions** from docDB (filter on
`session.*.software.name == "dynamic-foraging-task"` + `data_description.data_level == "derived"`),
**446 of 15822** `(subject_id, session_date)` day-groups have **more than one**
processed behavior asset (16338 sessions total). These are a **mix** of genuinely separate
same-day sessions and likely same-session artifacts, so a session table cannot safely be keyed by
`(subject_id, session_date)` alone — the full `session_name` (incl. `HH-MM-SS`) is needed as identity.

Han's pipeline session table, by contrast, has **at most one session per `(subject, date)`**, so this
is specific to the Code Ocean / docDB inventory.

## Sessions-per-(subject, date) distribution

| sessions in a day | # of (subject,date) |
|---|---|
| 1 | 15376 |
| 2 | 389 |
| 3 | 49 |
| 4 | 6 |
| 5 | 1 |
| 8 | 1 |

## Gap between same-day sessions (minutes)

min=0, 25%=6, median=23, 75%=50, max=738
- **< 5 min apart: 96 days** — likely the *same* session (restart / re-acquire / artifact)
- **>= 30 min apart: 184 days** — likely *genuinely separate* sessions (e.g. morning + afternoon)

## Examples

| subject | date | session times (HH-MM-SS) |
|---|---|---|
| `826162` | `2026-01-08` | 11-23-15, 12-46-16, 13-04-20, 13-16-57, 13-22-19, 13-24-59, 13-26-56, 13-28-41 |
| `712634` | `2024-05-29` | 11-10-52, 17-50-00, 19-36-02 |
| `731302` | `2024-10-08` | 13-59-25, 14-02-48 |

(e.g. the first is hours apart — clearly distinct sessions; the last is minutes apart — likely one session.)

## Why it matters

The foraging parquet-cache builder (`foraging_cache/`) unions Han's session table with the docDB/CO
universe. Matching is done on `(subject_id, session_date)` (Han's `nwb_suffix`/HHMMSS often differs
from docDB's for the same session), but sessions must **not** be collapsed by `(subject, date)` or the
~184+ genuinely-separate same-day sessions would be lost. Tracking here so we
decide how same-day duplicates (esp. the <5-min artifact pairs) should be handled / de-duplicated.

## Reproduce

```python
import pandas as pd, re
from aind_data_access_api.document_db import MetadataDbClient

client = MetadataDbClient(host="api.allenneuraldynamics.org",
database="metadata_index", collection="data_assets")
records = client.retrieve_docdb_records(
filter_query={
"$or": [
{"session.data_streams.software.name": "dynamic-foraging-task"},
{"session.stimulus_epochs.software.name": "dynamic-foraging-task"},
],
"data_description.data_level": "derived",
},
projection={"_id": 0, "name": 1},
paginate=True, paginate_batch_size=5000, # ~19k records; pagination required
)
df = pd.DataFrame(records)

# keep the behavior-processed NWB asset per session (drop PoseTracking / opto-sorted derived assets)
df["session_name"] = df["name"].str.extract(r"^(behavior_\d+_\d{4}-\d{2}-\d{2}_\d{2}-\d{2}-\d{2})_processed")
df = df.dropna(subset=["session_name"]).drop_duplicates("session_name")
df[["subj","date","tm"]] = df["session_name"].str.extract(r"behavior_(\d+)_(\d{4}-\d{2}-\d{2})_(\d{2}-\d{2}-\d{2})$")

per_day = df.groupby(["subj","date"]).size()
print(per_day.value_counts().sort_index()) # distribution
print(per_day[per_day > 1]) # the multi-session days
```

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.