AllenNeuralDynamics / AllenNeuralDynamics/aind-motion-energy

[Phase 6] M-1 · Emit processing.json across the pipeline (all three repos)

オープン
#20 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
主要言語
Python
スター
1
フォーク
0
平均マージ
2時間 27分
マージ済み PR(30日)
21

説明

Emit `processing.json` across the motion energy pipeline, per AIND metadata standards (`aind-data-schema`).

**This is the one item in the compliance plan retained without a citation on the [software practices page](https://docs.allenneuraldynamics.org/en/latest/policies_practices/software_practices.html)** — that page never mentions `processing.json` or provenance. It is kept because AIND metadata standards expect derived data assets to carry one, and this pipeline produces derived assets.

It is **deliberately sequenced last and filed as a single cross-repo issue** so its design can be settled later, with SciComp if needed, without blocking anything else. Nothing in the plan depends on M-1; M-1 depends only on L-9 (#12) for a version string to record.

Supersedes #11 (L-8, typed metadata model) and AllenNeuralDynamics/aind-motion-energy-capsule#3 (C-3), both closed into this issue.

### Implementation sketch

From `aind-data-schema` v2.9.0 (`examples/processing.py`):

```python
from aind_data_schema.components.identifiers import Code, DataAsset
from aind_data_schema.core.processing import (
DataProcess, Processing, ProcessName, ProcessStage,
)

p = Processing.create_with_sequential_process_graph(
pipelines=[Code(name=..., url=..., version=..., input_data=[DataAsset(name=...)])],
data_processes=[DataProcess(
process_type=ProcessName....,
stage=ProcessStage.PROCESSING,
code=Code(url=, version=, parameters=),
experimenters=[...],
start_date_time=..., end_date_time=...,
output_path=..., output_parameters=,
)],
)
p.write_standard_file(output_directory="/results")
```

### Open questions — resolve before implementing

- **`ProcessName` has no motion-energy member.** Closest are `VIDEO_ROI_TIMESERIES_EXTRACTION` and `OTHER` (which requires `name` or `notes` to be populated). Either use `OTHER` with a descriptive `name`, or propose a new member to `aind-data-schema-models`. **Good SciComp question.**
- **Which repo writes the file.** Per plan decision 3, provenance belongs in the capsule — but writing it needs Python, so the ME capsule regains a thin `code/run_capsule.py` alongside the CLI call, partially reversing capsule commit `1a86686`. If we go that way the new module must be **strictly provenance-only**: no compute, discovery, or plotting. The alternative is the batch launcher writing one record per fanned-out run, keeping the ME capsule a pure shell but putting provenance a layer away from the computation.
- **Whether the library's metadata `dict` needs a typed model.** L-8 assumed yes; it is not needed for `mypy --strict` (`dict[str, Any]` suffices), so the only justification would be M-1 wanting a validated `output_parameters` shape. Decide here. If a model is added, keep the serialized JSON key names identical so existing `_me_metadata.json` consumers are unaffected.
- **`aind-data-schema` version, and whether `Processing` should be appended to** rather than created fresh — the schema doc says the file "should be appended to with each subsequent stage of processing or analysis".
- **Requires a pinned capsule version to be meaningful** — see BA-2 (AllenNeuralDynamics/aind-motion-energy-batch#2).

### Repos touched

- `aind-motion-energy` — possibly a typed metadata model, if the open question above resolves that way
- `aind-motion-energy-capsule` — likely the writer
- `aind-motion-energy-batch` — capsule version pinning so the record is reproducible

Plan: [docs/standards-compliance.md](https://github.com/AllenNeuralDynamics/aind-motion-energy/blob/main/docs/standards-compliance.md#group-m--metadata)

*Size: L · Depends on: #12 · Blocks: nothing*

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。