Support for Databricks external locations beyond staging
- Dominant language
- Python
- Stars
- 5.9k
- Forks
- 600
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 38
Description
### Feature description
It would be great to allow for use of external locations outside of the typical staging layer. Right now, external locations are supported for staging, but upon table creation in Databricks Unity Catalog, all the source tables are "Managed". In a lakehouse architecture we would like the flexibility to use external locations for various use cases. This could be set at the source or resource level.
### Are you a dlt user?
Yes, I run dlt in production.
### Use case
In a hypothetical medallion architecture, we would have a landing, bronze, silver, and gold layer. We could have the staging layer set up to use external locations, specifically the landing layer. When we move from staging to the final destination, I want the destination table to use the bronze layer. Here is an example flow:
1 - Staging
`abfss://..dfs.core.windows.net/landing//`
in Unity catalog this would correlate to `..`
2 - Destination (not possible as of now)
`abfss://..dfs.core.windows.net/bronze//`
in Unity catalog this would correlate to `..`
### Proposed solution
To start it would be great to see this at the pipeline level. Here is what is currently in the documentation:
```
pipeline = dlt.pipeline(
pipeline_name='chess_pipeline',
destination='databricks',
staging=dlt.destinations.filesystem('abfss://@.dfs.core.windows.net/'), # add this to activate the staging location
dataset_name='player_data'
)
```
But ideally we would also want the ability to specify an external location for the destination data:
```
pipeline = dlt.pipeline(
pipeline_name='chess_pipeline',
destination='databricks',
staging=dlt.destinations.filesystem('abfss://@.dfs.core.windows.net/'), # add this to activate the staging location
external_loc=dlt.destinations.filesystem('abfss://@.dfs.core.windows.net/'),
dataset_name='player_data'
)
```
### Related issues
_No response_
Contributor guide
Research direction
Start by tracing the pipeline configuration shown in the issue and the Databricks destination handling for staging locations. Compare how destination tables are created as Managed and determine where a destination external location could be configured. Done means the proposed pipeline-level setting supports the bronze-path use case while preserving existing staging behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cloud, data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100