feat: declarative interface for SQL database
- Dominant language
- Python
- Stars
- 5.9k
- Forks
- 600
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 38
Description
## Summary
Provide a declarative interface to configure a SQL database source table-by-table (similar to REST API declarative API). Here's a mock API
```python
from dlt.sources.sql_database import sql_database
config = {
"tables": [
"customers", # defaults
{
"name": "items_resource", # resource name
"source_table_name": "inventory_items", table name in the source
"destination_table": "items",
},
{
"name": "orders",
"write_disposition": "merge",
"primary_key": "id",
"incremental": {
"cursor_path": "created_at",
"initial_value": "2024-01-25T00:00:00Z",
},
}
]
}
db_source = sql_database(config=config)
```
## Problem
Currently, it's easy to load all tables or select tables by name. It's not so easy to configure resources/tables deeply (e.g., incremental, primary key, write disposition). Users end-up creating custom "config to resource" interface.
Currently, there are 2 main approaches
### Apply hints on `sql_database` source
```python
from dlt.sources.sql_database import sql_database
# creates a connection, does reflection
db_source = sql_database()
# `load_config()` is an hypothetical function loading per-table config
for table_name, table_config in load_config().items():
db_source.resources[table_name].apply_hints(**table_config)
pipeline = dlt.pipeline(...)
pipeline.run(db_source)
```
Benefits: iterative process using `for` loops.
Downsides: instantiating `sql_database` is not free, it creates a connection to source; requires custom config loader format; `.apply_hints()` is a lesser known API; we're not validating the full config at once;
### Instantiate `sql_table` resources
```python
from dlt.sources.sql_database import sql_table
from dlt.sources.sql_database.helpers import engine_from_credentials
# create a connection to sql database source
@dlt.source
def my_custom_db_source():
engine = engine_from_credentials(...)
table_resources = [
# pass connection to avoid every resource creating a conn,
sql_table(credentials=engine, table="customers", schema="crm"),
sql_table(credentials=engine, table="orders", write_disposition="merge") ,
]
yield from table_resources
pipeline = dlt.pipeline(...)
pipeline.run(my_custom_db_source)
```
Benefits: explicit; Python native
Downsides: requires at least a single connection to database; the config is only valide per-table
## Potential solution
We could have a declarative API like the REST API sources: https://dlthub.com/docs/dlt-ecosystem/verified-sources/rest_api/basic#endpoint-configuration This would be faster and ligher to parse and compile.
This would be great for human and agent collaboration with everything explicit and readable
## Next steps
- Identify limitations of current approach
- Design the API before implementation, the param names should match the REST API source
- Making the interface JSON serializable (e.g., dictionary with basic types) would allow easy integration with YAML, TOML, and agents generating specs.
Contributor guide
Assessment
This issue has not been assessed yet.