dlt-hub / dlt-hub/dlt

Support for Databricks external locations beyond staging

Open
#1,979 4 comments 3 reactions 0 assignees View on GitHub
credentials
Dominant language
Python
Stars
5.9k
Forks
600
Avg merge
1d 14h
Merged PRs (30d)
38

Description

### Feature description

It would be great to allow for use of external locations outside of the typical staging layer. Right now, external locations are supported for staging, but upon table creation in Databricks Unity Catalog, all the source tables are "Managed". In a lakehouse architecture we would like the flexibility to use external locations for various use cases. This could be set at the source or resource level.

### Are you a dlt user?

Yes, I run dlt in production.

### Use case

In a hypothetical medallion architecture, we would have a landing, bronze, silver, and gold layer. We could have the staging layer set up to use external locations, specifically the landing layer. When we move from staging to the final destination, I want the destination table to use the bronze layer. Here is an example flow:

1 - Staging
`abfss://..dfs.core.windows.net/landing//`
in Unity catalog this would correlate to `..`

2 - Destination (not possible as of now)
`abfss://..dfs.core.windows.net/bronze//`
in Unity catalog this would correlate to `..`

### Proposed solution

To start it would be great to see this at the pipeline level. Here is what is currently in the documentation:

```
pipeline = dlt.pipeline(
pipeline_name='chess_pipeline',
destination='databricks',
staging=dlt.destinations.filesystem('abfss://@.dfs.core.windows.net/'), # add this to activate the staging location
dataset_name='player_data'
)
```
But ideally we would also want the ability to specify an external location for the destination data:

```
pipeline = dlt.pipeline(
pipeline_name='chess_pipeline',
destination='databricks',
staging=dlt.destinations.filesystem('abfss://@.dfs.core.windows.net/'), # add this to activate the staging location
external_loc=dlt.destinations.filesystem('abfss://@.dfs.core.windows.net/'),
dataset_name='player_data'
)
```

### Related issues

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by tracing the pipeline configuration shown in the issue and the Databricks destination handling for staging locations. Compare how destination tables are created as Managed and determine where a destination external location could be configured. Done means the proposed pipeline-level setting supports the bronze-path use case while preserving existing staging behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cloud, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.