AbsaOSS / AbsaOSS/spline-spark-agent

Write lineage metadata to centralized file on hive

Open
#856 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
210
Forks
102
Avg merge
1d 1h
Merged PRs (30d)
1

Description

Hi @wajda
I used on the HDFS on cloudera this config
spark.jars=hdfs:///tmp/spark-2.4-spline-agent-bundle_2.11-2.2.1.jar
spark.sql.queryExecutionListeners=za.co.absa.spline.harvester.listener.SplineQueryExecutionListener
spark.spline.mode=ENABLED
spark.spline.lineageDispatcher=hdfs
spark.spline.lineageDispatcher.hdfs.outputDir=hdfs:///tmp/spline/lineage/
spark.spline.lineageDispatcher.hdfs.fileNamePrefix=lineage_
spark.spline.lineageDispatcher.hdfs.fileBufferSize=4096
spark.spline.lineageDispatcher.hdfs.filePermissions=777
spark.driver.memory=4g

But it wrote the lineage to the target file on the script, seems to ignore the spark.spline.lineageDispatcher.hdfs.outputDir=hdfs:///tmp/spline/lineage/
Is there a way to set it to a centralized place and also to give each json file the execution_plan_id
thanks

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.