AllenNeuralDynamics / AllenNeuralDynamics/aind-software-docs

Document the S3 / DocDB / Code Ocean data-asset contract and consistency expectations

Offen
#217 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Python
Sterne
1
Forks
0
Ø Merge
1 T. 6 Std.
Gemergte PRs (30 T.)
7

Beschreibung

**Is your feature request related to a problem? Please describe.**

A dataset is tracked in three places — S3 (the bytes), DocDB (the metadata record), and Code Ocean (catalog entries) — but no page states how they relate or what consistency to expect. Reconciling across systems requires reverse-engineering it. Undocumented behaviors hit in a recent cleanup audit:

- Missing counterpart: data in S3 with no DocDB record or no CO asset; v1 records never upgraded to v2.
- Dangling reference: records pointing at S3 locations or CO assets that no longer exist.
- Duplication: multiple CO assets for one S3 path (indexer minting a second, provenance-less asset against a prefix still being written); multiple DocDB records sharing one name.
- Incorrect content: wrong data level, invalid metadata.

**Describe the solution you'd like**

A dedicated page to describe: which system is authoritative for what, who writes each and when (transfer-service, pipeline capture, indexer), a description of how consistently tracked data-assets may appear, and the known inconsistency modes above.

**Describe alternatives you've considered**

-

**Additional context**

A documented contract should define the components well enough that validators can be built to check cross-source consistency

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.