cdklabs / cdklabs/cdk-pipelines-github

Recursion into cdc.out on GitHubWorkflow pipeline if .dockerignore does not exclude CDK files/directories

Open
#927 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
TypeScript
Stars
384
Forks
45
Avg merge
1d 1h
Merged PRs (30d)
4

Description

Not sure if this is a bug but it took me ages to work out a fix and I thought I'd share.

unless .dockerignore includes the CDK related directories then a GitHubWorkflow pipeline recurses the entire application in cdk.out then crashes

This works on a normal CDK synth/deploy however in a GitHubWorkflow pipeline it recurses the app into the cdk.out directory and then crashes

I'm deploying a lambda from a DockerImageFunction using a pipeline

```
image = _lambda.DockerImageCode.from_image_asset(
file=f"{constants.SERVICE}/src/Dockerfile",
directory=".",
cmd=[f"{constants.SERVICE}.src.core.{constants.SERVICE}._handler.lambda_handler"],
platform=Platform.LINUX_AMD64,
asset_name=constants.SERVICE,
)

lambda_id = f"{id_}-lambda"

lambda_fn: _lambda.DockerImageFunction = _lambda.DockerImageFunction(
self,
id=lambda_id,
function_name=f"{constants.SERVICE}-event-handler",
architecture=_lambda.Architecture.X86_64,
code=image
)
```

If I add the following to the .dockerignore then there is no recursion and everything works as it did without pipelines
```
# CDK dir
cdk # cdk iac code & tests dir
cdk.out # cdk build dir

# CDK Files
cdk.context.json
cdk.json
app.py
```

without that when the CDK hits the lambda_fn code=image line it crashes (see below)

~/PycharmProjects/nextgen-ingest development +23 !15 ?25 ❯ make aws-synth 16s  xxxt-X-_Snmio 16:43:31
cdk synth --app="python3 xxxt/app.py" --require-approval=never --profile xxx
jsii.errors.JavaScriptError:
Error: ENAMETOOLONG: name too long, copyfile '/xxx/PycharmProjects/nextgen-ingest/cdk.out/asset.9368f33f85b336115178717344aaf3dab03de137511199f60e6268cdffca5d7a/cdk.out/asset.9368f33f85b336115178717344aaf3dab03de137511199f60e6268cdffca5d7a/cdk.out/asset.9368f33f85b336115178717344aaf3dab03de137511199f60e6268cdffca5d7a/cdk.out/asset.9368f33f85b336115178717344aaf3dab03de137511199f60e6268cdffca5d7a/cdk.out/asset.9368f33f85b336115178717344aaf3dab03de137511199f60e6268cdffca5d7a/cdk.out/asset.9368f33f85b336115178717344aaf3dab03de137511199f60e6268cdffca5d7a/cdk.out/asset.9368f33f85b336115178717344aaf3dab03de137511199f60e6268cdffca5d7a/cdk.out/asset.9368f33f85b336115178717344aaf3dab03de137511199f60e6268cdffca5d7a/cdk.out/asset.9368f33f85b336115178717344aaf3dab03de137511199f60e6268cdffca5d7a/cdk.out/asset.152e97b507e729d4de48a50c1703e2c92e9f4e3cf285b7af6f3d85e9c9b28920/' -> '/xxx/PycharmProjects/nextgen-ingest/cdk.out/asset.9368f33f85b336115178717344aaf3dab03de137511199f60e6268cdffca5d7a/cdk.out/asset.9368f33f85b336115178717344aaf3dab03de137511199f60e6268cdffca5d7a/cdk.out/asset.9368f33f85b336115178717344aaf3dab03de137511199f60e6268cdffca5d7a/cdk.out/asset.9368f33f85b336115178717344aaf3dab03de137511199f60e6268cdffca5d7a/cdk.out/asset.9368f33f85b336115178717344aaf3dab03de137511199f60e6268cdffca5d7a/cdk.out/asset.9368f33f85b336115178717344aaf3dab03de137511199f60e6268cdffca5d7a/cdk.out/asset.9368f33f85b336115178717344aaf3dab03de137511199f60e6268cdffca5d7a/cdk.out/asset.9368f33f85b336115178717344aaf3dab03de137511199f60e6268cdffca5d7a/cdk.out/asset.9368f33f85b336115178717344aaf3dab03de137511199f60e6268cdffca5d7a/cdk.out/asset.9368f33f85b336115178717344aaf3dab03de137511199f60e6268cdffca5d7a/cdk.out/asset.152e97b507e729d4de48a50c1703e2c92e9f4e3cf285b7af6f3d85e9c9b28920/'
at Object.copyFileSync (node:fs:3029:11)
at copyDirectory (/var/folders/9_/zp0mvc257454b0ds2qfhghqm0000gn/T/jsii-kernel-Xu3VkL/node_modules/aws-cdk-lib/core/lib/fs/copy.js:1:1499)
at copyDirectory (/var/folders/9_/zp0mvc257454b0ds2qfhghqm0000gn/T/jsii-kernel-Xu3VkL/node_modules/aws-cdk-lib/core/lib/fs/copy.js:1:1400)
at copyDirectory (/var/folders/9_/zp0mvc257454b0ds2qfhghqm0000gn/T/jsii-kernel-Xu3VkL/node_modules/aws-cdk-lib/core/lib/fs/copy.js:1:1400)
at copyDirectory (/var/folders/9_/zp0mvc257454b0ds2qfhghqm0000gn/T/jsii-kernel-Xu3VkL/node_modules/aws-cdk-lib/core/lib/fs/copy.js:1:1400)
at copyDirectory (/var/folders/9_/zp0mvc257454b0ds2qfhghqm0000gn/T/jsii-kernel-Xu3VkL/node_modules/aws-cdk-lib/core/lib/fs/copy.js:1:1400)
at copyDirectory (/var/folders/9_/zp0mvc257454b0ds2qfhghqm0000gn/T/jsii-kernel-Xu3VkL/node_modules/aws-cdk-lib/core/lib/fs/copy.js:1:1400)
at copyDirectory (/var/folders/9_/zp0mvc257454b0ds2qfhghqm0000gn/T/jsii-kernel-Xu3VkL/node_modules/aws-cdk-lib/core/lib/fs/copy.js:1:1400)
at copyDirectory (/var/folders/9_/zp0mvc257454b0ds2qfhghqm0000gn/T/jsii-kernel-Xu3VkL/node_modules/aws-cdk-lib/core/lib/fs/copy.js:1:1400)
at copyDirectory (/var/folders/9_/zp0mvc257454b0ds2qfhghqm0000gn/T/jsii-kernel-Xu3VkL/node_modules/aws-cdk-lib/core/lib/fs/copy.js:1:1400)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/xxx/PycharmProjects/nextgen-ingest/app.py", line 19, in
dev_stage = IngestPipelineStage(app, "dev", env=env_DEV)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/xxx/.local/share/virtualenvs/nextgen-ingest-X-_Snmio/lib/python3.12/site-packages/jsii/_runtime.py", line 118, in __call__
inst = super(JSIIMeta, cast(JSIIMeta, cls)).__call__(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/xxx/PycharmProjects/nextgen-ingest/cdk/src/iac/pipeline_stage.py", line 14, in __init__
ingest_stack = IngestStack(
^^^^^^^^^^^^
File "/xxx/.local/share/virtualenvs/nextgen-ingest-X-_Snmio/lib/python3.12/site-packages/jsii/_runtime.py", line 118, in __call__
inst = super(JSIIMeta, cast(JSIIMeta, cls)).__call__(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/xxx/PycharmProjects/nextgen-ingest/cdk/src/services/ingest_stack.py", line 25, in __init__
self.ingest_lambda = IngestLambdaConstruct(
^^^^^^^^^^^^^^^^^^^^^^
File "/xxx/.local/share/virtualenvs/nextgen-ingest-X-_Snmio/lib/python3.12/site-packages/jsii/_runtime.py", line 118, in __call__
inst = super(JSIIMeta, cast(JSIIMeta, cls)).__call__(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/xxx/PycharmProjects/nextgen-ingest/cdk/src/services/ingest_construct.py", line 33, in __init__
self.create_ingest_func = self._build_lambda(
^^^^^^^^^^^^^^^^^^^
File "/xxx/PycharmProjects/nextgen-ingest/cdk/src/services/ingest_construct.py", line 167, in _build_lambda
lambda_fn: _lambda.DockerImageFunction = _lambda.DockerImageFunction(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/xxx/.local/share/virtualenvs/nextgen-ingest-X-_Snmio/lib/python3.12/site-packages/jsii/_runtime.py", line 118, in __call__
inst = super(JSIIMeta, cast(JSIIMeta, cls)).__call__(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/xxx/.local/share/virtualenvs/nextgen-ingest-X-_Snmio/lib/python3.12/site-packages/aws_cdk/aws_lambda/__init__.py", line 26627, in __init__
jsii.create(self.__class__, self, [scope, id, props])
File "/xxx/.local/share/virtualenvs/nextgen-ingest-X-_Snmio/lib/python3.12/site-packages/jsii/_kernel/__init__.py", line 334, in create
response = self.provider.create(
^^^^^^^^^^^^^^^^^^^^^
File "/xxx/.local/share/virtualenvs/nextgen-ingest-X-_Snmio/lib/python3.12/site-packages/jsii/_kernel/providers/process.py", line 365, in create
return self._process.send(request, CreateResponse)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/xxx/.local/share/virtualenvs/nextgen-ingest-X-_Snmio/lib/python3.12/site-packages/jsii/_kernel/providers/process.py", line 342, in send
raise RuntimeError(resp.error) from JavaScriptError(resp.stack)
RuntimeError: ENAMETOOLONG: name too long, copyfile '/xxx/PycharmProjects/nextgen-ingest/cdk.out/asset.9368f33f85b336115178717344aaf3dab03de137511199f60e6268cdffca5d7a/cdk.out/asset.9368f33f85b336115178717344aaf3dab03de137511199f60e6268cdffca5d7a/cdk.out/asset.9368f33f85b336115178717344aaf3dab03de137511199f60e6268cdffca5d7a/cdk.out/asset.9368f33f85b336115178717344aaf3dab03de137511199f60e6268cdffca5d7a/cdk.out/asset.9368f33f85b336115178717344aaf3dab03de137511199f60e6268cdffca5d7a/cdk.out/asset.9368f33f85b336115178717344aaf3dab03de137511199f60e6268cdffca5d7a/cdk.out/asset.9368f33f85b336115178717344aaf3dab03de137511199f60e6268cdffca5d7a/cdk.out/asset.9368f33f85b336115178717344aaf3dab03de137511199f60e6268cdffca5d7a/cdk.out/asset.9368f33f85b336115178717344aaf3dab03de137511199f60e6268cdffca5d7a/cdk.out/asset.152e97b507e729d4de48a50c1703e2c92e9f4e3cf285b7af6f3d85e9c9b28920/' -> '/xxx/PycharmProjects/nextgen-ingest/cdk.out/asset.9368f33f85b336115178717344aaf3dab03de137511199f60e6268cdffca5d7a/cdk.out/asset.9368f33f85b336115178717344aaf3dab03de137511199f60e6268cdffca5d7a/cdk.out/asset.9368f33f85b336115178717344aaf3dab03de137511199f60e6268cdffca5d7a/cdk.out/asset.9368f33f85b336115178717344aaf3dab03de137511199f60e6268cdffca5d7a/cdk.out/asset.9368f33f85b336115178717344aaf3dab03de137511199f60e6268cdffca5d7a/cdk.out/asset.9368f33f85b336115178717344aaf3dab03de137511199f60e6268cdffca5d7a/cdk.out/asset.9368f33f85b336115178717344aaf3dab03de137511199f60e6268cdffca5d7a/cdk.out/asset.9368f33f85b336115178717344aaf3dab03de137511199f60e6268cdffca5d7a/cdk.out/asset.9368f33f85b336115178717344aaf3dab03de137511199f60e6268cdffca5d7a/cdk.out/asset.9368f33f85b336115178717344aaf3dab03de137511199f60e6268cdffca5d7a/cdk.out/asset.152e97b507e729d4de48a50c1703e2c92e9f4e3cf285b7af6f3d85e9c9b28920/'

Subprocess exited with error 1
make: *** [Makefile:45: aws-synth] Error 1

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.