AbsaOSS / AbsaOSS/spline-spark-agent

Lineage Calculation not triggered when using Spark Connector for SQL Server BulkCopy API

Offen
#58 3 Kommentare 0 Reaktionen 1 zugewiesene Person Beansprucht von @cerveada Auf GitHub ansehen
feature
Vorherrschende Sprache
Scala
Sterne
210
Forks
102
Ø Merge
1 T. 1 Std.
Gemergte PRs (30 T.)
1

Beschreibung

When using the Spark Sql Server connector (https://github.com/Azure/azure-sqldb-spark), there are two ways to produce the final output:

One way the Spark SQL Connectior API works is by using the usual df.write style operation which triggers lineage via the "Save" action (see example below)

However, when calling df.bulkCopyToSqlDB (also shown below), no lineage is triggered as no action is produced at all by the bulkCopyToSqlDB method as it doesn't go through the usual .jdbc chain. Is there a way to force Spline to produce the lineage for the bulkCopyToSqlDB "terminal" operation?

val df = ...//Read from wherever

//Using the Spark Sql Server connector
val writeConfig = Config(Map(
"url" -> "databaseservername",
"databaseName" -> "catalogname",
"dbTable" -> "tablename",
"user" -> "user",
"password" -> "password"
))

//This is the Spark Sql Server connector API using the usual write technique.
df.write.mode(SaveMode.Append).sqlDB(writeConfig) //Lineage gets triggered

//Spark Sql Connector also allows for a custom bulk copy that doesn't go through .jdbc
df.bulkCopyToSqlDB(writeConfig) //No action and as a result no lineage

Thanks,
Harish.

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.