AbsaOSS / AbsaOSS/spline-spark-agent
Spline doesn't track some in memory operations in pyspark
- Ngôn ngữ chính
- Scala
- Star
- 210
- Fork
- 102
- Merge trung bình
- 1 ngày 1 giờ
- Pull request đã merge (30 ngày)
- 1
Mô tả
I found that spline in pyspark doesn't track some in memory operations like collect, head and toPandas.
Operations count and show are tracked as expected.
I used spline with bundle-3.2 in test:
https://mvnrepository.com/artifact/za.co.absa.spline.agent.spark/spark-3.2-spline-agent-bundle_2.12/2.0.0
Here is my pyspark options:
`JAVA_HOME=/Users/alexey.balyshev/Library/Java/JavaVirtualMachines/corretto-1.8.0_402/Contents/Home/ ~/spark-3.2.2-bin-hadoop3.2/bin/pyspark --master local --deploy-mode client --jars ~/Documents/spark-3.2-spline-agent-bundle_2.12-2.0.0.jar --num-executors 1 --conf "spark.executor.cores=1" --conf "spark.sql.queryExecutionListeners=za.co.absa.spline.harvester.listener.SplineQueryExecutionListener" --conf "spark.spline.lineageDispatcher=console" --conf "spark.spline.plugins.za.co.absa.spline.harvester.plugin.embedded.NonPersistentActionsCapturePlugin.enabled=true"`
After show and count operations input I could see the execution plan in json format, but after collect, head and toPandas operations I got an empty output.
At the same time in spark-shell all in memory operations are tracked as expected.
Hướng dẫn đóng góp
Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này
Đánh giá
Issue này chưa được đánh giá.