AbsaOSS / AbsaOSS/spline-spark-agent
Support for AWS Glue DynamicFrame
- 主要言語
- Scala
- スター
- 210
- フォーク
- 102
- 平均マージ
- 1日 1時間
- マージ済み PR(30日)
- 1
説明
When adding the Spline agent bundle to an AWS Glue Python script (Spark 3.3, Python 3), lineage is produced when using the standard patterns like `df = spark.read.csv(file_path, header=True, inferSchema=True) and df.write...` as expected.
However, AWS Glue does have a concept of [Dynamic Frames](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-crawler-pyspark-extensions-dynamic-frame.html) usage of which which looks something like
```python
import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
args = getResolvedOptions(sys.argv, ["JOB_NAME"])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args["JOB_NAME"], args)
# Script generated for node Amazon S3
f0 = glueContext.create_dynamic_frame.from_options(
format_options={
"quoteChar": -1,
"withHeader": True,
"separator": ",",
"optimizePerformance": False,
},
connection_type="s3",
format="csv",
connection_options={
## Need to replace with S3 path with file name
"paths": ["s3://...."],
"recurse": True,
},
transformation_ctx="f0",
)
# Script generated for node Amazon S3
f1 = glueContext.write_dynamic_frame.from_options(
frame=f0,
connection_type="s3",
format="csv",
connection_options={
## Need to replace with S3 path
"path": "s3://....",
"partitionKeys": [],
},
transformation_ctx="f1",
)
job.commit()
```
Can Spline support this dynamic frame pattern in AWS Glue? I used the spark-3.3-spline-agent-bundle_2.12-2.0.0.jar bundle - Spline agent initialized successfully, but could not produce lineage.
コントリビューションガイド
このリポジトリのコントリビューションガイドは索引されていません
評価
この issue はまだ評価されていません。