AllenNeuralDynamics / AllenNeuralDynamics/aind-software-docs

Document the S3 / DocDB / Code Ocean data-asset contract and consistency expectations

Aberta
#217 1 comentário 0 reações 0 responsáveis Ver no GitHub
Linguagem predominante
Python
Estrelas
1
Forks
0
Merge médio
1d 6h
PRs com merge (30d)
7

Descrição

**Is your feature request related to a problem? Please describe.**

A dataset is tracked in three places — S3 (the bytes), DocDB (the metadata record), and Code Ocean (catalog entries) — but no page states how they relate or what consistency to expect. Reconciling across systems requires reverse-engineering it. Undocumented behaviors hit in a recent cleanup audit:

- Missing counterpart: data in S3 with no DocDB record or no CO asset; v1 records never upgraded to v2.
- Dangling reference: records pointing at S3 locations or CO assets that no longer exist.
- Duplication: multiple CO assets for one S3 path (indexer minting a second, provenance-less asset against a prefix still being written); multiple DocDB records sharing one name.
- Incorrect content: wrong data level, invalid metadata.

**Describe the solution you'd like**

A dedicated page to describe: which system is authoritative for what, who writes each and when (transfer-service, pipeline capture, indexer), a description of how consistently tracked data-assets may appear, and the known inconsistency modes above.

**Describe alternatives you've considered**

-

**Additional context**

A documented contract should define the components well enough that validators can be built to check cross-source consistency

Guia de contribuição

Nenhum guia de contribuição indexado para este repositório

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.