AbsaOSS / AbsaOSS/spline-spark-agent

Support for AWS Glue DynamicFrame

Offen
#781 2 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
feature
Vorherrschende Sprache
Scala
Sterne
210
Forks
102
Ø Merge
1 T. 1 Std.
Gemergte PRs (30 T.)
1

Beschreibung

When adding the Spline agent bundle to an AWS Glue Python script (Spark 3.3, Python 3), lineage is produced when using the standard patterns like `df = spark.read.csv(file_path, header=True, inferSchema=True) and df.write...` as expected.

However, AWS Glue does have a concept of [Dynamic Frames](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-crawler-pyspark-extensions-dynamic-frame.html) usage of which which looks something like

```python
import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job

args = getResolvedOptions(sys.argv, ["JOB_NAME"])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args["JOB_NAME"], args)

# Script generated for node Amazon S3
f0 = glueContext.create_dynamic_frame.from_options(
format_options={
"quoteChar": -1,
"withHeader": True,
"separator": ",",
"optimizePerformance": False,
},
connection_type="s3",
format="csv",
connection_options={
## Need to replace with S3 path with file name
"paths": ["s3://...."],
"recurse": True,
},
transformation_ctx="f0",
)

# Script generated for node Amazon S3
f1 = glueContext.write_dynamic_frame.from_options(
frame=f0,
connection_type="s3",
format="csv",
connection_options={
## Need to replace with S3 path
"path": "s3://....",
"partitionKeys": [],
},
transformation_ctx="f1",
)

job.commit()
```
Can Spline support this dynamic frame pattern in AWS Glue? I used the spark-3.3-spline-agent-bundle_2.12-2.0.0.jar bundle - Spline agent initialized successfully, but could not produce lineage.

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.