cdklabs / cdklabs/cdk-stacksets

Unable to deploy Glue job assets

Open
#300 2 comments 0 reactions 0 assignees View on GitHub
bug effort/medium p2
Dominant language
TypeScript
Stars
118
Forks
25
Avg merge
33m
Merged PRs (30d)
4

Description

I am trying to create a stack set containing a Glue job, I have a conflict in how the script asset is handled by the Job construct and how the stackset assets are managed during the deployment.

When using the Job construct the code asset expects to receive a single file.

```
from aws_cdk import aws_glue_alpha as glue
from cdk_stacksets import StackSetStack

dirname = os.path.dirname(__file__)

class GlueJobStack(StackSetStack):

def __init__(self, scope: Construct, id: str, **kwargs) -> None:
super().__init__(scope, id, **kwargs)

...

glue_job = glue.Job(
self,
"GlueJob",
job_name="glue_job",
executable=glue.JobExecutable.python_etl(
glue_version=glue.GlueVersion.V3_0,
python_version=glue.PythonVersion.THREE,
script=glue.Code.from_asset(
path=os.path.join(dirname,
"glue_jobs/script.py"
),
deploy_time=True,
)
),
worker_type=glue.WorkerType.G_1_X,
worker_count=10,
role=glue_role
)

```

When executing the `cdk synth` command I get the error: `Asset path must be either a .zip file or a directory`

If I try using a path to the `glue_jobs` folder instead I get the error: `Code path ./glue_jobs/ is a directory. Only files are supported`. This is because the `glue.Code.from_asset` method expects a single file, not a directory.

Another approach:

```
job_asset = assets.Asset(
self,
"JobAsset",
path=os.path.join(dirname,
"glue_jobs/script.py"
)
)

...

```

I have tried using an `Asset` construct first but I get the same `Asset path must be either a .zip file or a directory` error if my asset refers to a single file. Again, if I reference the folder containing the python job script and in the job definition `glue.Code. from_bucket` instead, the synth and deploy process works but the job is created using the zip file (binary file) as the source so the code is illegible and can't be executed.

Contributor guide

Open the contributing guide

Research direction

Start with the glue.Code.from_asset and assets.Asset usage shown in the issue, then reproduce the failure with cdk synth for a StackSet containing a Glue job. Trace how the single-file script and directory assets are packaged and passed to the Glue job; done means the StackSet deploys a readable, executable Glue script rather than a zip binary.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.