AllenNeuralDynamics / AllenNeuralDynamics/aind-software-docs

Document the S3 / DocDB / Code Ocean data-asset contract and consistency expectations

未關閉
#217 1 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
主要語言
Python
星號
1
分支
0
平均合併
1 天 6 小時
30 天內合併 PR
7

描述

**Is your feature request related to a problem? Please describe.**

A dataset is tracked in three places — S3 (the bytes), DocDB (the metadata record), and Code Ocean (catalog entries) — but no page states how they relate or what consistency to expect. Reconciling across systems requires reverse-engineering it. Undocumented behaviors hit in a recent cleanup audit:

- Missing counterpart: data in S3 with no DocDB record or no CO asset; v1 records never upgraded to v2.
- Dangling reference: records pointing at S3 locations or CO assets that no longer exist.
- Duplication: multiple CO assets for one S3 path (indexer minting a second, provenance-less asset against a prefix still being written); multiple DocDB records sharing one name.
- Incorrect content: wrong data level, invalid metadata.

**Describe the solution you'd like**

A dedicated page to describe: which system is authoritative for what, who writes each and when (transfer-service, pipeline capture, indexer), a description of how consistently tracked data-assets may appear, and the known inconsistency modes above.

**Describe alternatives you've considered**

-

**Additional context**

A documented contract should define the components well enough that validators can be built to check cross-source consistency

貢獻指南

這個儲存庫沒有索引到貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。