AllenNeuralDynamics / AllenNeuralDynamics/aind-motion-energy
[Phase 6] M-1 · Emit processing.json across the pipeline (all three repos)
- 主要言語
- Python
- スター
- 1
- フォーク
- 0
- 平均マージ
- 2時間 27分
- マージ済み PR(30日)
- 21
説明
Emit `processing.json` across the motion energy pipeline, per AIND metadata standards (`aind-data-schema`).
**This is the one item in the compliance plan retained without a citation on the [software practices page](https://docs.allenneuraldynamics.org/en/latest/policies_practices/software_practices.html)** — that page never mentions `processing.json` or provenance. It is kept because AIND metadata standards expect derived data assets to carry one, and this pipeline produces derived assets.
It is **deliberately sequenced last and filed as a single cross-repo issue** so its design can be settled later, with SciComp if needed, without blocking anything else. Nothing in the plan depends on M-1; M-1 depends only on L-9 (#12) for a version string to record.
Supersedes #11 (L-8, typed metadata model) and AllenNeuralDynamics/aind-motion-energy-capsule#3 (C-3), both closed into this issue.
### Implementation sketch
From `aind-data-schema` v2.9.0 (`examples/processing.py`):
```python
from aind_data_schema.components.identifiers import Code, DataAsset
from aind_data_schema.core.processing import (
DataProcess, Processing, ProcessName, ProcessStage,
)
p = Processing.create_with_sequential_process_graph(
pipelines=[Code(name=..., url=..., version=..., input_data=[DataAsset(name=...)])],
data_processes=[DataProcess(
process_type=ProcessName....,
stage=ProcessStage.PROCESSING,
code=Code(url=, version=, parameters=),
experimenters=[...],
start_date_time=..., end_date_time=...,
output_path=..., output_parameters=,
)],
)
p.write_standard_file(output_directory="/results")
```
### Open questions — resolve before implementing
- **`ProcessName` has no motion-energy member.** Closest are `VIDEO_ROI_TIMESERIES_EXTRACTION` and `OTHER` (which requires `name` or `notes` to be populated). Either use `OTHER` with a descriptive `name`, or propose a new member to `aind-data-schema-models`. **Good SciComp question.**
- **Which repo writes the file.** Per plan decision 3, provenance belongs in the capsule — but writing it needs Python, so the ME capsule regains a thin `code/run_capsule.py` alongside the CLI call, partially reversing capsule commit `1a86686`. If we go that way the new module must be **strictly provenance-only**: no compute, discovery, or plotting. The alternative is the batch launcher writing one record per fanned-out run, keeping the ME capsule a pure shell but putting provenance a layer away from the computation.
- **Whether the library's metadata `dict` needs a typed model.** L-8 assumed yes; it is not needed for `mypy --strict` (`dict[str, Any]` suffices), so the only justification would be M-1 wanting a validated `output_parameters` shape. Decide here. If a model is added, keep the serialized JSON key names identical so existing `_me_metadata.json` consumers are unaffected.
- **`aind-data-schema` version, and whether `Processing` should be appended to** rather than created fresh — the schema doc says the file "should be appended to with each subsequent stage of processing or analysis".
- **Requires a pinned capsule version to be meaningful** — see BA-2 (AllenNeuralDynamics/aind-motion-energy-batch#2).
### Repos touched
- `aind-motion-energy` — possibly a typed metadata model, if the open question above resolves that way
- `aind-motion-energy-capsule` — likely the writer
- `aind-motion-energy-batch` — capsule version pinning so the record is reproducible
Plan: [docs/standards-compliance.md](https://github.com/AllenNeuralDynamics/aind-motion-energy/blob/main/docs/standards-compliance.md#group-m--metadata)
*Size: L · Depends on: #12 · Blocks: nothing*
コントリビューションガイド
評価
この issue はまだ評価されていません。