cdk-pipelines: pipelines.ConfirmPermissionsBroadening not skipping changes without Permission Broadening
- Dominant language
- TypeScript
- Stars
- 12.9k
- Forks
- 4.6k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 83
Description
### Describe the bug
In the past if the cdk diff didnt show any security changes the ConfirmPermissionsBroadening (CodeBuild Action) would call the CDKPipelinesAutoApprove Lambda Function which automatically would approve the Lambda Function. This is not happening anymore and a manual approval is always required.
We noticed that this feature stopped working for us in multiple indepent Accounts and Repositories without any updates (cdk-Version ,Bootstrapping etc.) on our end. The last time that we saw the step working as expected was November 4th.
### Regression Issue
- [ ] Select this option if this issue appears to be a regression.
### Last Known Working CDK Version
_No response_
### Expected Behavior
I would expect that the CDK diff detects that there are no changes and calls the Lambda Function. This was the behaviour in the past. The Logs of the CodeBuild look like this:
```text
[Container] 2024/11/04 16:54:49.319085 Running on CodeBuild On-demand
[Container] 2024/11/04 16:54:49.319098 Waiting for agent ping
[Container] 2024/11/04 16:54:49.520531 Waiting for DOWNLOAD_SOURCE
[Container] 2024/11/04 16:54:50.777909 Phase is DOWNLOAD_SOURCE
[Container] 2024/11/04 16:54:50.815397 CODEBUILD_SRC_DIR=/codebuild/output/srcXXXXXXX/src
[Container] 2024/11/04 16:54:50.815950 YAML location is /codebuild/readonly/buildspec.yml
[Container] 2024/11/04 16:54:50.819309 Setting HTTP client timeout to higher timeout for S3 source
[Container] 2024/11/04 16:54:50.819422 Processing environment variables
[Container] 2024/11/04 16:54:51.007034 No runtime version selected in buildspec.
[Container] 2024/11/04 16:54:51.059621 Moving to directory /codebuild/output/srcXXXXXXX/src
[Container] 2024/11/04 16:54:51.061258 Unable to initialize cache download: no paths specified to be cached
[Container] 2024/11/04 16:54:51.178746 Configuring ssm agent with target id: codebuild:XXXXXXX
[Container] 2024/11/04 16:54:51.179779 Successfully updated ssm agent configuration
[Container] 2024/11/04 16:54:51.180166 Registering with agent
[Container] 2024/11/04 16:54:51.216167 Phases found in YAML: 1
[Container] 2024/11/04 16:54:51.216237 BUILD: 11 commands
[Container] 2024/11/04 16:54:51.216572 Phase complete: DOWNLOAD_SOURCE State: SUCCEEDED
[Container] 2024/11/04 16:54:51.216585 Phase context status code: Message:
[Container] 2024/11/04 16:54:51.285608 Entering phase INSTALL
[Container] 2024/11/04 16:54:51.319698 Phase complete: INSTALL State: SUCCEEDED
[Container] 2024/11/04 16:54:51.319714 Phase context status code: Message:
[Container] 2024/11/04 16:54:51.355941 Entering phase PRE_BUILD
[Container] 2024/11/04 16:54:51.361266 Phase complete: PRE_BUILD State: SUCCEEDED
[Container] 2024/11/04 16:54:51.361281 Phase context status code: Message:
[Container] 2024/11/04 16:54:51.396937 Entering phase BUILD
[Container] 2024/11/04 16:54:51.433396 Running command npm install -g aws-cdk
added 1 package in 9s
[Container] 2024/11/04 16:55:06.075130 Running command export PIPELINE_NAME="$(node -pe '`${process.env.CODEBUILD_INITIATOR}`.split("/")[1]')"
[Container] 2024/11/04 16:55:06.110247 Running command payload="$(node -pe 'JSON.stringify({ "PipelineName": process.env.PIPELINE_NAME, "StageName": process.env.STAGE_NAME, "ActionName": process.env.ACTION_NAME })' )"
[Container] 2024/11/04 16:55:06.143489 Running command ARN=$CODEBUILD_BUILD_ARN
[Container] 2024/11/04 16:55:06.148583 Running command REGION="$(node -pe '`${process.env.ARN}`.split(":")[3]')"
[Container] 2024/11/04 16:55:06.181179 Running command ACCOUNT_ID="$(node -pe '`${process.env.ARN}`.split(":")[4]')"
[Container] 2024/11/04 16:55:06.214130 Running command PROJECT_NAME="$(node -pe '`${process.env.ARN}`.split(":")[5].split("/")[1]')"
[Container] 2024/11/04 16:55:06.247115 Running command PROJECT_ID="$(node -pe '`${process.env.ARN}`.split(":")[6]')"
[Container] 2024/11/04 16:55:06.279713 Running command export LINK="https://$REGION.console.aws.amazon.com/codesuite/codebuild/$ACCOUNT_ID/projects/$PROJECT_NAME/build/$PROJECT_NAME:$PROJECT_ID/?region=$REGION"
[Container] 2024/11/04 16:55:06.322494 Running command export PIPELINE_LINK="https://$REGION.console.aws.amazon.com/codesuite/codepipeline/pipelines/$PIPELINE_NAME/view?region=$REGION"
[Container] 2024/11/04 16:55:06.327848 Running command if cdk diff -a . --security-only --fail $STAGE_PATH/\*; then aws lambda invoke --function-name MyPipeline-PipelinePipelinesSecurit-tjblx3z6zu4g3 --invocation-type Event --cli-binary-format raw-in-base64-out --payload "$payload" lambda.out; export MESSAGE="No security-impacting changes detected."; else [ -z "${NOTIFICATION_ARN}" ] || aws sns publish --topic-arn $NOTIFICATION_ARN --subject "$NOTIFICATION_SUBJECT" --message "An upcoming change would broaden security changes in $PIPELINE_NAME.
Review and approve the changes in CodePipeline to proceed with the deployment.
Review the changes in CodeBuild:
$LINK
Approve the changes in CodePipeline (stage $STAGE_NAME, action $ACTION_NAME):
$PIPELINE_LINK"; export MESSAGE="Deployment would make security-impacting changes. Click the link below to inspect them, then click Approve if all changes are expected."; fi
current credentials could not be used to assume 'arn:aws:iam::012345678901:role/cdk-cgh650dgw-lookup-role-012345678901-eu-central-1', but are for the right account. Proceeding anyway.
Lookup role exists but was not assumed. Proceeding with default credentials.
Lookup role exists but was not assumed. Proceeding with default credentials.
start: Building ckMp6awVd6qv9dKdgTJE3DBZcy0CjwNdWryeX7HaR4C48kjQD8Q4CGp6N0EW3RJF:012345678901-eu-central-1
success: Built ckMp6awVd6qv9dKdgTJE3DBZcy0CjwNdWryeX7HaR4C48kjQD8Q4CGp6N0EW3RJF:012345678901-eu-central-1
start: Publishing ckMp6awVd6qv9dKdgTJE3DBZcy0CjwNdWryeX7HaR4C48kjQD8Q4CGp6N0EW3RJF:012345678901-eu-central-1
current credentials could not be used to assume 'arn:aws:iam::012345678901:role/cdk-cgh650dgw-file-publishing-role-012345678901-eu-central-1', but are for the right account. Proceeding anyway.
fail: Bucket named 'cdk-cgh650dgw-assets-012345678901-eu-central-1' exists, but we dont have access to it.
Could not create a change set, will base the diff on template differences (run again with -v to see the reason)
Stack Pipeline/Pipepline/MyStack (MyStack)
✨ Number of stacks with differences: 0
[Container] 2024/11/04 16:55:20.275824 Phase complete: BUILD State: SUCCEEDED
[Container] 2024/11/04 16:55:20.275843 Phase context status code: Message:
[Container] 2024/11/04 16:55:20.307476 Entering phase POST_BUILD
[Container] 2024/11/04 16:55:20.312519 Phase complete: POST_BUILD State: SUCCEEDED
[Container] 2024/11/04 16:55:20.312536 Phase context status code: Message:
[Container] 2024/11/04 16:55:20.355916 Set report auto-discover timeout to 5 seconds
[Container] 2024/11/04 16:55:20.355993 Expanding base directory path: .
[Container] 2024/11/04 16:55:20.357593 Assembling file list
[Container] 2024/11/04 16:55:20.357608 Expanding .
[Container] 2024/11/04 16:55:20.359236 Expanding file paths for base directory .
[Container] 2024/11/04 16:55:20.359247 Assembling file list
[Container] 2024/11/04 16:55:20.359251 Expanding **/*
[Container] 2024/11/04 16:55:20.361148 No matching auto-discover report paths found
[Container] 2024/11/04 16:55:20.361165 Report auto-discover file discovery took 0.005249 seconds
[Container] 2024/11/04 16:55:20.361191 Phase complete: UPLOAD_ARTIFACTS State: SUCCEEDED
[Container] 2024/11/04 16:55:20.361197 Phase context status code: Message:
```
### Current Behavior
Currently the CDK diff always fails which causes the Lambda to not be triggered. The Logs of the CodeBuild look like this:
```text
[Container] 2024/12/10 12:42:52.550106 Running on CodeBuild On-demand
[Container] 2024/12/10 12:42:52.550117 Waiting for agent ping
[Container] 2024/12/10 12:42:52.651078 Waiting for DOWNLOAD_SOURCE
[Container] 2024/12/10 12:42:53.677653 Phase is DOWNLOAD_SOURCE
[Container] 2024/12/10 12:42:53.712676 CODEBUILD_SRC_DIR=/codebuild/output/srcXXXXX/src
[Container] 2024/12/10 12:42:53.713236 YAML location is /codebuild/readonly/buildspec.yml
[Container] 2024/12/10 12:42:53.717394 Setting HTTP client timeout to higher timeout for S3 source
[Container] 2024/12/10 12:42:53.717516 Processing environment variables
[Container] 2024/12/10 12:42:53.940972 No runtime version selected in buildspec.
[Container] 2024/12/10 12:42:53.959129 Moving to directory /codebuild/output/srcXXXX/src
[Container] 2024/12/10 12:42:53.960662 Unable to initialize cache download: no paths specified to be cached
[Container] 2024/12/10 12:42:54.060708 Configuring ssm agent with target id: codebuild:XXXXXX
[Container] 2024/12/10 12:42:54.061492 Successfully updated ssm agent configuration
[Container] 2024/12/10 12:42:54.061825 Registering with agent
[Container] 2024/12/10 12:42:54.095927 Phases found in YAML: 1
[Container] 2024/12/10 12:42:54.095942 BUILD: 11 commands
[Container] 2024/12/10 12:42:54.096324 Phase complete: DOWNLOAD_SOURCE State: SUCCEEDED
[Container] 2024/12/10 12:42:54.096336 Phase context status code: Message:
[Container] 2024/12/10 12:42:54.163936 Entering phase INSTALL
[Container] 2024/12/10 12:42:54.203398 Phase complete: INSTALL State: SUCCEEDED
[Container] 2024/12/10 12:42:54.203414 Phase context status code: Message:
[Container] 2024/12/10 12:42:54.239046 Entering phase PRE_BUILD
[Container] 2024/12/10 12:42:54.244181 Phase complete: PRE_BUILD State: SUCCEEDED
[Container] 2024/12/10 12:42:54.244194 Phase context status code: Message:
[Container] 2024/12/10 12:42:54.280869 Entering phase BUILD
[Container] 2024/12/10 12:42:54.315902 Running command npm install -g aws-cdk
added 1 package in 8s
[Container] 2024/12/10 12:43:07.462099 Running command export PIPELINE_NAME="$(node -pe '`${process.env.CODEBUILD_INITIATOR}`.split("/")[1]')"
[Container] 2024/12/10 12:43:07.494760 Running command payload="$(node -pe 'JSON.stringify({ "PipelineName": process.env.PIPELINE_NAME, "StageName": process.env.STAGE_NAME, "ActionName": process.env.ACTION_NAME })' )"
[Container] 2024/12/10 12:43:07.525573 Running command ARN=$CODEBUILD_BUILD_ARN
[Container] 2024/12/10 12:43:07.530026 Running command REGION="$(node -pe '`${process.env.ARN}`.split(":")[3]')"
[Container] 2024/12/10 12:43:07.560674 Running command ACCOUNT_ID="$(node -pe '`${process.env.ARN}`.split(":")[4]')"
[Container] 2024/12/10 12:43:07.592502 Running command PROJECT_NAME="$(node -pe '`${process.env.ARN}`.split(":")[5].split("/")[1]')"
[Container] 2024/12/10 12:43:07.630729 Running command PROJECT_ID="$(node -pe '`${process.env.ARN}`.split(":")[6]')"
[Container] 2024/12/10 12:43:07.662314 Running command export LINK="https://$REGION.console.aws.amazon.com/codesuite/codebuild/$ACCOUNT_ID/projects/$PROJECT_NAME/build/$PROJECT_NAME:$PROJECT_ID/?region=$REGION"
[Container] 2024/12/10 12:43:07.666499 Running command export PIPELINE_LINK="https://$REGION.console.aws.amazon.com/codesuite/codepipeline/pipelines/$PIPELINE_NAME/view?region=$REGION"
[Container] 2024/12/10 12:43:07.670348 Running command if cdk diff -a . --security-only --fail $STAGE_PATH/\*; then aws lambda invoke --function-name Pipeline-PipelinePipelinesSecurit-XXXXX --invocation-type Event --cli-binary-format raw-in-base64-out --payload "$payload" lambda.out; export MESSAGE="No security-impacting changes detected."; else [ -z "${NOTIFICATION_ARN}" ] || aws sns publish --topic-arn $NOTIFICATION_ARN --subject "$NOTIFICATION_SUBJECT" --message "An upcoming change would broaden security changes in $PIPELINE_NAME.
Review and approve the changes in CodePipeline to proceed with the deployment.
Review the changes in CodeBuild:
$LINK
Approve the changes in CodePipeline (stage $STAGE_NAME, action $ACTION_NAME):
$PIPELINE_LINK"; export MESSAGE="Deployment would make security-impacting changes. Click the link below to inspect them, then click Approve if all changes are expected."; fi
current credentials could not be used to assume 'arn:aws:iam::012345678901:role/cdk-cgh650dgw-lookup-role-012345678901-eu-central-1', but are for the right account. Proceeding anyway.
Lookup role arn:aws:iam::012345678901:role/cdk-cgh650dgw-lookup-role-012345678901-eu-central-1 was not assumed. Proceeding with default credentials.
User: arn:aws:sts::012345678901:assumed-role/Pipeline-PipelinePipelinesSecurity-XXXX/AWSCodeBuild-XXXXXX is not authorized to perform: cloudformation:DescribeStacks on resource: arn:aws:cloudformation:eu-central-1:012345678901:stack/MyStack/* because no identity-based policy allows the cloudformation:DescribeStacks action
[Container] 2024/12/10 12:43:09.086719 Phase complete: BUILD State: SUCCEEDED
[Container] 2024/12/10 12:43:09.086737 Phase context status code: Message:
[Container] 2024/12/10 12:43:09.125476 Entering phase POST_BUILD
[Container] 2024/12/10 12:43:09.130289 Phase complete: POST_BUILD State: SUCCEEDED
[Container] 2024/12/10 12:43:09.130305 Phase context status code: Message:
[Container] 2024/12/10 12:43:09.178086 Set report auto-discover timeout to 5 seconds
[Container] 2024/12/10 12:43:09.178184 Expanding base directory path: .
[Container] 2024/12/10 12:43:09.179824 Assembling file list
[Container] 2024/12/10 12:43:09.179838 Expanding .
[Container] 2024/12/10 12:43:09.181468 Expanding file paths for base directory .
[Container] 2024/12/10 12:43:09.181477 Assembling file list
[Container] 2024/12/10 12:43:09.181481 Expanding **/*
[Container] 2024/12/10 12:43:09.183695 Found 1 file(s)
[Container] 2024/12/10 12:43:09.183764 Report auto-discover file discovery took 0.005679 seconds
[Container] 2024/12/10 12:43:09.184395 Phase complete: UPLOAD_ARTIFACTS State:
```
It seems like creating it without the IAM Permissions based on template differences isnt working anymore. I checked via Cloud-Trail that there were no Changes to the IAM Permissions of the Role. Original this was deployed with package version `aws-cdk-lib==2.151.0`, later I updatet to `aws-cdk-lib==2.163.0` and `aws-cdk-lib==2.164.0`. But there was no change
### Reproduction Steps
Create a Pipeline with Manual Permission Check. Rough Example of what this looks like:
```python
code_pipeline = pipelines.CodePipeline(
self,
"Pipeline",
code_build_defaults=pipelines.CodeBuildOptions(
build_environment=aws_codebuild.BuildEnvironment(
build_image=aws_codebuild.LinuxBuildImage.AMAZON_LINUX_2_5,
compute_type=aws_codebuild.ComputeType.SMALL,
),
),
synth=pipelines.ShellStep(
"Synth",
input=pipelines.CodePipelineSource.connection(
repo_string=repo_string,
branch="master",
action_name="Source",
connection_arn=connection_arn,
),
commands=[
"pip install -r requirements.txt",
"npm install -g aws-cdk",
"cdk synth",
],
),
)
the_stage = PipelineStage(self, id, account_var=account_var, env=env)
code_pipeline.add_stage(
stage=the_stage,
pre=[
pipelines.ConfirmPermissionsBroadening(
"PermissionCheck",
stage=the_stage,
)
],
)
```
### Possible Solution
If I manually add these Permissions to the Role of CDKSecurityCheck and than it starts working:
```text
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"cloudformation:ListStacks",
"cloudformation:DescribeStacks",
"cloudformation:GetTemplate"
],
"Resource": "*"
}
]
}
```
But since the Pipeline are self Mutating this manual change just get its overwritten. I found https://github.com/aws/aws-cdk/issues/27605#issuecomment-1819737786, but im not able to access `Pipeline/PipelinesSecurityCheck/CDKSecurityCheck/Role` in this way since the Construct doesnt expose the Node.
It would be great if somebody could show me how to do the same as done in https://github.com/aws/aws-cdk/issues/27605#issuecomment-1819737786 but for the Construct, so I can apply my workarround. Or if the underlying bug (?) gets fixed.
### Additional Information/Context
_No response_
### CDK CLI Version
2.163.1 (build c14a1ff)
### Framework Version
_No response_
### Node.js Version
20
### OS
Amazon Linux 2023
### Language
Python
### Language Version
3.9.20
### Other information
_No response_
Contributor guide
Research direction
Start by reproducing the CodeBuild command `cdk diff -a . --security-only --fail ...` shown in the logs and inspect the implementation of `pipelines.ConfirmPermissionsBroadening`, including its CDKPipelinesAutoApprove Lambda path. Compare the no-security-change and permission-broadening outcomes; done means the no-change case invokes the Lambda instead of requiring manual approval, with a regression test if the repository provides one.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, typescript
- Domain
- ci-cd, cloud, devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100