Change validation of files (assets) to align with validation on the server
- Dominant language
- Python
- Stars
- 28
- Forks
- 37
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 9
Description
The underlying trigger for the issue is
- https://github.com/dandi/dandi-schema/issues/284
where assets started to fail validation, preventing publishing of dandisets, only when already on the archive. It boiled down to the fails in validation against `jsonschema` export of pydantic models which is done to guarantee that meditor validation could be done online (thus in JS), and the difference in treating `date-time` format by jsonschema and pydantic (datetime).
Looking at the code, in dandi-archive we do
```python
from dandischema.metadata import aggregate_assets_summary, validate
...
metadata = asset.published_metadata()
validate(metadata, schema_key='PublishedAsset', json_validation=True)
```
while in the dandi-cli we do not use `dandischema..validate` construct and rather
```
try:
asset = self.get_metadata(digest=self._DUMMY_DIGEST)
BareAsset(**asset.model_dump())
except ValidationError as e:
if devel_debug:
raise
return _pydantic_errors_to_validation_results(
e, self.filepath, scope=Scope.FILE
)
```
so in effect only validating against Pydantic (and for a BareAsset).
edit 1: moreover we do not even get to above from NWBAsset.get_validation_errors due to
```python
if schema_version is not None:
errors.extend(
super().get_validation_errors(
schema_version=schema_version, devel_debug=devel_debug
)
)
else:
# make sure that we have some basic metadata fields we require
try:
origin = ValidationOrigin(
name="nwbinspector",
version=str(_get_nwb_inspector_version()),
)
for error in inspect_nwbfile(
...
```
so we solely rely on nwbinspector inspection for NWB file but that one does not care about `dandischema` "directly". So we might want to change that one way or another...
We should harmonize, and I feel that here we should use `dandischema.metadata.validate`. I think it also aligns better with our desire to separate models from intended "validity" assumptions.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.