Running dlt in Databricks
- Dominant language
- Python
- Stars
- 5.9k
- Forks
- 600
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 38
Description
### Documentation description
I am trying to use dlt within a Databricks environment.
However, the dlt import fails due to a name clash with the Databricks' delta live tables.
Issue detailed here https://dlthub.notion.site/Databricks-notebook-instructions-980832a90fab4a98b6c8aa010d47646e.
Also tried the init script tweak from this reddit thread relating to the above guide. Didn't work.
https://www.reddit.com/r/dataengineering/comments/1g2wl3k/using_data_load_tool_dlthub_together_with/
However, the workaround here does not work.
There is no surfaced error message or log, just that the init script fails.
I would like the guide linked above to be updated or replaced.
### Are you a dlt user?
Yes, I'm already a dlt user.
Contributor guide
Research direction
Start with the linked Databricks notebook instructions and the referenced init-script workaround, then reproduce the dlt import name clash and the init-script failure while collecting available logs. Done means the guide is updated or replaced with a verified Databricks procedure and explains how to resolve the conflict.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, documentation
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100