cdklabs / cdklabs/cdk-stacksets
Unable to deploy Glue job assets
- Dominant language
- TypeScript
- Stars
- 118
- Forks
- 25
- Avg merge
- 33m
- Merged PRs (30d)
- 4
Description
I am trying to create a stack set containing a Glue job, I have a conflict in how the script asset is handled by the Job construct and how the stackset assets are managed during the deployment.
When using the Job construct the code asset expects to receive a single file.
```
from aws_cdk import aws_glue_alpha as glue
from cdk_stacksets import StackSetStack
dirname = os.path.dirname(__file__)
class GlueJobStack(StackSetStack):
def __init__(self, scope: Construct, id: str, **kwargs) -> None:
super().__init__(scope, id, **kwargs)
...
glue_job = glue.Job(
self,
"GlueJob",
job_name="glue_job",
executable=glue.JobExecutable.python_etl(
glue_version=glue.GlueVersion.V3_0,
python_version=glue.PythonVersion.THREE,
script=glue.Code.from_asset(
path=os.path.join(dirname,
"glue_jobs/script.py"
),
deploy_time=True,
)
),
worker_type=glue.WorkerType.G_1_X,
worker_count=10,
role=glue_role
)
```
When executing the `cdk synth` command I get the error: `Asset path must be either a .zip file or a directory`
If I try using a path to the `glue_jobs` folder instead I get the error: `Code path ./glue_jobs/ is a directory. Only files are supported`. This is because the `glue.Code.from_asset` method expects a single file, not a directory.
Another approach:
```
job_asset = assets.Asset(
self,
"JobAsset",
path=os.path.join(dirname,
"glue_jobs/script.py"
)
)
...
```
I have tried using an `Asset` construct first but I get the same `Asset path must be either a .zip file or a directory` error if my asset refers to a single file. Again, if I reference the folder containing the python job script and in the job definition `glue.Code. from_bucket` instead, the synth and deploy process works but the job is created using the zip file (binary file) as the source so the code is illegible and can't be executed.
Contributor guide
Research direction
Start with the glue.Code.from_asset and assets.Asset usage shown in the issue, then reproduce the failure with cdk synth for a StackSet containing a Glue job. Trace how the single-file script and directory assets are packaged and passed to the Glue job; done means the StackSet deploys a readable, executable Glue script rather than a zip binary.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 50/100