AbsaOSS / AbsaOSS/spline-spark-agent
Support for AWS Glue DynamicFrame
- Ngôn ngữ chính
- Scala
- Star
- 210
- Fork
- 102
- Merge trung bình
- 1 ngày 1 giờ
- Pull request đã merge (30 ngày)
- 1
Mô tả
When adding the Spline agent bundle to an AWS Glue Python script (Spark 3.3, Python 3), lineage is produced when using the standard patterns like `df = spark.read.csv(file_path, header=True, inferSchema=True) and df.write...` as expected.
However, AWS Glue does have a concept of [Dynamic Frames](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-crawler-pyspark-extensions-dynamic-frame.html) usage of which which looks something like
```python
import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
args = getResolvedOptions(sys.argv, ["JOB_NAME"])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args["JOB_NAME"], args)
# Script generated for node Amazon S3
f0 = glueContext.create_dynamic_frame.from_options(
format_options={
"quoteChar": -1,
"withHeader": True,
"separator": ",",
"optimizePerformance": False,
},
connection_type="s3",
format="csv",
connection_options={
## Need to replace with S3 path with file name
"paths": ["s3://...."],
"recurse": True,
},
transformation_ctx="f0",
)
# Script generated for node Amazon S3
f1 = glueContext.write_dynamic_frame.from_options(
frame=f0,
connection_type="s3",
format="csv",
connection_options={
## Need to replace with S3 path
"path": "s3://....",
"partitionKeys": [],
},
transformation_ctx="f1",
)
job.commit()
```
Can Spline support this dynamic frame pattern in AWS Glue? I used the spark-3.3-spline-agent-bundle_2.12-2.0.0.jar bundle - Spline agent initialized successfully, but could not produce lineage.
Hướng dẫn đóng góp
Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này
Đánh giá
Issue này chưa được đánh giá.