AbsaOSS / AbsaOSS/spline-spark-agent
Databricks - Support of Delta live tables
- Dominant language
- Scala
- Stars
- 210
- Forks
- 102
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 1
Description
I have a case of using delta live tables that runs SQL notebooks.
I cannot add cell of python
`sc._jvm.za.co.absa.spline.harvester.SparkLineageInitializer.enableLineageTracking(spark._jsparkSession)`
since DLT doesn't support magic like `%python`
So I created a separate notebook of python - but still fails - although the spline Jar is installed on the cluster
I think that because DLT manages dynamically the clusters - I think that all the installation of jars on the cluster take no effect.
## Question
Does someone has experience using spline in Databricks in DLT?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.