dlt-hub / dlt-hub/dlt

Specify schema (for schema contracts) as JSON schema

Open
#4,271 0 comments 0 reactions 1 assignee Claimed by @rudolfix View on GitHub
Feature needs design
Dominant language
Python
Stars
5.9k
Forks
600
Avg merge
1d 14h
Merged PRs (30d)
38

Description

### Feature description

Currently, you can use Pydantic models or json objects to specify the schema that a data source should adhere to. It would be good to be able to use a valid JSON schema standard to specify the schema (as well as import/export schemas) instead of a non-standard homebrew format. This would make it a lot easier to integrate dlt with resources/destinations that already have schemas available in JSON Schema or where there are converters to/from JSON Schema.

Depending on how validation is implemented under the hood, it might also be possible to integrate JSON schema validation into pipelines for "free".

### Are you a dlt user?

Yes, I run dlt in production.

### Use case

Running pipelines using data sources that provide JSON Schema for their data or destinations where the schema is available as JSON Schema.

### Proposed solution

Ideally, migrate from the current homebrew way of specifying a schema to using valid JSON Schema. Pydantic models can be created from JSON Schema, so that is a possible way in. For my own use cases, I would prefer to not have overhead of converting back and forth between JSON and Pydantic objects, so adapting the existing schema validation/contract enforcement code to use valid JSON schema instead of the homebrew schema would be best.

### Related issues

Couldn't find any.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.