dandi / dandi/dandi-schema

Overhaul models so that "unfinished" metadata can be represented without cheating Pydantic

Open
#205 22 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7
Forks
15
PR merge metrics
No merged PRs in 30d

Description

**Note:** @satra filed a spiritually-identical issue at almost the same time: #204

---

(This is an accumulation of various things discussed elsewhere which need to be written down.)

Currently, it is possible for users of the Dandi Archive API to submit asset & Dandiset metadata that does not fully conform to the relevant dandischema model, and the Archive will accept, store, and return such flawed metadata, largely via use of `DandiBaseModel.unvalidated` (being replaced by Pydantic's `construct` in #203). I believe part of the motivation for this is so that web users can fill in metadata over multiple sessions without having to fill in every field in a single sitting.

This results in API requests for asset & Dandiset metadata sometimes returning values that do not validate under dandischema's models; in particular, if a user of dandi-cli's Python API calls `get_metadata()` instead of `get_raw_metadata()`, the call may fail because our API returned metadata that doesn't conform to our own models (See dandi/dandi-cli#1205 and
dandi/dandi-cli#1363).

The dandischema models should therefore be overhauled as follows:

* There should exist models for representing Dandiset & asset metadata in a draft/unfinished state. These models should accept all inputs that we want to accept from users (both via the API and the web UI), store in the database, and return in API responses. (It is likely that such models will have all of their fields marked optional aside from the absolute bare minimum required.)
* The `get_metadata()` methods of dandi-cli's Python API should return instances of these models.

* There should exist functionality for determining whether an instance of a draft/unfinished model meets all of the requirements for Dandiset publication.
* One possible way to implement this would be to have separate models for published metadata that inherit from the draft models and make various fields non-optional.

CC @satra @dandi/dandiarchive @dandi/dandi-cli

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.