EnergySystemsModellingLab / EnergySystemsModellingLab/MUSE2

Use `frictionless` to validate schemas and CSV files

Open
#623 1 comment 0 reactions 1 assignee Claimed by @alexdewar View on GitHub
documentation
Dominant language
Rust
Stars
8
Forks
5
Avg merge
1d 2m
Merged PRs (30d)
37

Description

We have schema files in the table schema format in `schemas/input` which are used primarily to generate documentation for the input format. We will also use a similar mechanism for the output format (#529).

We could use the Python-based [`frictionless`](https://framework.frictionlessdata.io/) framework to validate all the CSV files for the example models. We're already checking that the files are well-formed with our Rust tests, so this would be primarily to check that the schemas themselves are well-formed (which is kind of the opposite to how they're normally used). It seems you can either do this from the command-line or from a Python script. We should run it on the CI to enforce this.

There seems to be a concept in `frictionless` called "packages" which I don't entirely understand, but it may be a way we can group all these schemas together somehow (the only way I could get it to work was by running individual schemas against individual files, but that's a bit gross and doesn't enforce file names).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.