AllenNeuralDynamics / AllenNeuralDynamics/aind-software-docs
Document the S3 / DocDB / Code Ocean data-asset contract and consistency expectations
- Vorherrschende Sprache
- Python
- Sterne
- 1
- Forks
- 0
- Ø Merge
- 1 T. 6 Std.
- Gemergte PRs (30 T.)
- 7
Beschreibung
**Is your feature request related to a problem? Please describe.**
A dataset is tracked in three places — S3 (the bytes), DocDB (the metadata record), and Code Ocean (catalog entries) — but no page states how they relate or what consistency to expect. Reconciling across systems requires reverse-engineering it. Undocumented behaviors hit in a recent cleanup audit:
- Missing counterpart: data in S3 with no DocDB record or no CO asset; v1 records never upgraded to v2.
- Dangling reference: records pointing at S3 locations or CO assets that no longer exist.
- Duplication: multiple CO assets for one S3 path (indexer minting a second, provenance-less asset against a prefix still being written); multiple DocDB records sharing one name.
- Incorrect content: wrong data level, invalid metadata.
**Describe the solution you'd like**
A dedicated page to describe: which system is authoritative for what, who writes each and when (transfer-service, pipeline capture, indexer), a description of how consistently tracked data-assets may appear, and the known inconsistency modes above.
**Describe alternatives you've considered**
-
**Additional context**
A documented contract should define the components well enough that validators can be built to check cross-source consistency
Beitragsleitfaden
Für dieses Repository ist kein Beitragsleitfaden indexiert
Bewertung
Dieses Issue wurde noch nicht bewertet.