Specify schema (for schema contracts) as JSON schema
- Dominant language
- Python
- Stars
- 5.9k
- Forks
- 600
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 38
Description
### Feature description
Currently, you can use Pydantic models or json objects to specify the schema that a data source should adhere to. It would be good to be able to use a valid JSON schema standard to specify the schema (as well as import/export schemas) instead of a non-standard homebrew format. This would make it a lot easier to integrate dlt with resources/destinations that already have schemas available in JSON Schema or where there are converters to/from JSON Schema.
Depending on how validation is implemented under the hood, it might also be possible to integrate JSON schema validation into pipelines for "free".
### Are you a dlt user?
Yes, I run dlt in production.
### Use case
Running pipelines using data sources that provide JSON Schema for their data or destinations where the schema is available as JSON Schema.
### Proposed solution
Ideally, migrate from the current homebrew way of specifying a schema to using valid JSON Schema. Pydantic models can be created from JSON Schema, so that is a possible way in. For my own use cases, I would prefer to not have overhead of converting back and forth between JSON and Pydantic objects, so adapting the existing schema validation/contract enforcement code to use valid JSON schema instead of the homebrew schema would be best.
### Related issues
Couldn't find any.
Contributor guide
Assessment
This issue has not been assessed yet.