dlt-hub / dlt-hub/dlt

feat: declarative interface for SQL database

Open
#4,337 2 comments 1 reaction 1 assignee Claimed by @zilto View on GitHub
enhancement needs design
Dominant language
Python
Stars
5.9k
Forks
600
Avg merge
1d 14h
Merged PRs (30d)
38

Description

## Summary
Provide a declarative interface to configure a SQL database source table-by-table (similar to REST API declarative API). Here's a mock API

```python
from dlt.sources.sql_database import sql_database

config = {
"tables": [
"customers", # defaults
{
"name": "items_resource", # resource name
"source_table_name": "inventory_items", table name in the source
"destination_table": "items",
},
{
"name": "orders",
"write_disposition": "merge",
"primary_key": "id",
"incremental": {
"cursor_path": "created_at",
"initial_value": "2024-01-25T00:00:00Z",
},
}
]
}

db_source = sql_database(config=config)
```

## Problem
Currently, it's easy to load all tables or select tables by name. It's not so easy to configure resources/tables deeply (e.g., incremental, primary key, write disposition). Users end-up creating custom "config to resource" interface.

Currently, there are 2 main approaches

### Apply hints on `sql_database` source

```python
from dlt.sources.sql_database import sql_database

# creates a connection, does reflection
db_source = sql_database()

# `load_config()` is an hypothetical function loading per-table config
for table_name, table_config in load_config().items():
db_source.resources[table_name].apply_hints(**table_config)

pipeline = dlt.pipeline(...)
pipeline.run(db_source)
```

Benefits: iterative process using `for` loops.

Downsides: instantiating `sql_database` is not free, it creates a connection to source; requires custom config loader format; `.apply_hints()` is a lesser known API; we're not validating the full config at once;

### Instantiate `sql_table` resources

```python
from dlt.sources.sql_database import sql_table
from dlt.sources.sql_database.helpers import engine_from_credentials

# create a connection to sql database source

@dlt.source
def my_custom_db_source():
engine = engine_from_credentials(...)
table_resources = [
# pass connection to avoid every resource creating a conn,
sql_table(credentials=engine, table="customers", schema="crm"),
sql_table(credentials=engine, table="orders", write_disposition="merge") ,
]
yield from table_resources

pipeline = dlt.pipeline(...)
pipeline.run(my_custom_db_source)
```
Benefits: explicit; Python native

Downsides: requires at least a single connection to database; the config is only valide per-table

## Potential solution
We could have a declarative API like the REST API sources: https://dlthub.com/docs/dlt-ecosystem/verified-sources/rest_api/basic#endpoint-configuration This would be faster and ligher to parse and compile.

This would be great for human and agent collaboration with everything explicit and readable

## Next steps
- Identify limitations of current approach
- Design the API before implementation, the param names should match the REST API source
- Making the interface JSON serializable (e.g., dictionary with basic types) would allow easy integration with YAML, TOML, and agents generating specs.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.