AbsaOSS / AbsaOSS/spline-spark-agent
Spline doest report lineage to hdfs
- 主要言語
- Scala
- スター
- 210
- フォーク
- 102
- 平均マージ
- 1日 1時間
- マージ済み PR(30日)
- 1
説明
## Describe the bug
I put on the config of spark
spark.jars=hdfs:///tmp/spark-2.4-spline-agent-bundle_2.11-2.2.1.jar
spark.sql.queryExecutionListeners=za.co.absa.spline.harvester.listener.SplineQueryExecutionListener
spark.spline.mode=ENABLED
spark.spline.lineageDispatcher=hdfs
spark.spline.lineageDispatcher.hdfs.outputDir=hdfs:///tmp/spline/lineage/
spark.spline.lineageDispatcher.hdfs.fileNamePrefix=lineage_
spark.spline.lineageDispatcher.hdfs.fileBufferSize=4096
spark.spline.lineageDispatcher.hdfs.filePermissions=777
spark.driver.memory=4g
I run this code
from pyspark.sql import SparkSession
spark = SparkSession.builder \
.appName("Write DataFrame to HDFS as CSV") \
.getOrCreate()
# Create a sample DataFrame
data = [
(1, "Alice", 28),
(2, "Bob", 24),
(3, "Cathy", 29)
]
columns = ["Id", "Name", "Age"]
df = spark.createDataFrame(data, columns)
df.show()
output_path = "hdfs:///tmp/sample_data"
df.coalesce(1).write \
.option("header", True) \
.mode("overwrite")\
.csv(output_path)
print("DataFrame written to HDFS at {}".format(output_path))
spark.stop()
the log show that all ok
But when i go to
[root@ccycloud-2.nightly7x-us-ol.root.comops.site Scripts]# hdfs dfs -ls /tmp/spline/lineage/
I see no data
attached the log
## Versions
Please provide versions of: Spline, Spark and Scala that were in use when the bug happened.
Spark- 2.4.8.7.2.18.0-641
/_/
Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_232)
[Spark 2 4 Spline Agent Bundle](https://mvnrepository.com/artifact/za.co.absa.spline.agent.spark/spark-2.4-spline-agent-bundle) » [2.2.1](https://mvnrepository.com/artifact/za.co.absa.spline.agent.spark/spark-2.4-spline-agent-bundle_2.12/2.2.1)
コントリビューションガイド
このリポジトリのコントリビューションガイドは索引されていません
評価
この issue はまだ評価されていません。