dandi / dandi/dandi-cli

Support multiple versions of target dandischema

Open
#1,571 1 comment 0 reactions 0 assignees View on GitHub
schema vendoring
Dominant language
Python
Stars
28
Forks
37
Avg merge
1d 17h
Merged PRs (30d)
9

Description

Inspired by

- https://github.com/dandi/dandi-cli/issues/1463

by @kabilar , and ongoing work on "vendorization" of DANDI archive codebase to support multiple instances of DANDI.

ATM both `dandi-cli` and `dandi-archive` re-use `dandischema` Python library for manipulating and validating metadata records. As a result, any upgrade of `dandischema` in server requires upgrade of `dandischema` in `dandi-cli` installation which also might require upgrade of `dandi-cli` itself.

With multiple archive instances in play (already DANDI and LINC and EMBER is coming), synchronizing all of them to align with most recent `dandischema` would be getting close to infeasible. So I think we should distill a solution which would allow `dandi-cli` to manipulate different/needed versions of dandischema.

Possible roadmap to achieve that
- migrate pydantic model to linkml (effort ongoing by @candleindark ) + potentially additional "instance specific" restrictions (TODO: link issues in dandi-schema)
- export linkml models into https://github.com/dandi/schema/tree/master/releases
- define `dandischema_schema_version` -- as long as it is "compatible" in terms of its formulation with `dandi-cli`. In a similar situation in BIDS we have `BIDS_VERSION` and its `SCHEMA_VERSION`, see https://github.com/bids-standard/bids-specification/tree/master/src/schema#version-of-the-schema
- in `dandi-cli` implement loading of models from `linkml` models for the version desired by the server:
- as long as that `linkml` model is described in compatible version of schema (based on minimal `dandischema_schema_version` supported by dandi-cli) , we should be good
- potentially extend the model with instance specific models where needed for validation (most likely the base model would be sufficient to describe)
- not yet sure what we should do for currently used pydantic models in Python interfaces, might need to populate classes for different versions. So issue similarish to what we discussed with @rly and other nwb developers about nwb versions.

Also attn @jwodder @satra @bendichter @dandi/dandiarchive to potentially also think about enhancement to above plan or just to suggest alternative approaches.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.